2024-09-16T10:19:24.853 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T10:19:24.871 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T10:19:24.969 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655 branch: reef description: rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906655' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ mon_bind_msgr2: false sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1390 sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 sleep_before_teardown: 0 subset: 5319/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 targets: smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMEbhcnxotVjRvNXM4TQfLe90v6fM8oMlstd1MndhTfk6Ee+NUOdeja0vhjcKFi+2jKG7g1E5Z98W6dJ1yZWkGI= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjIz7uKVJ6cOLwfgH8Ei+hjNgGTTYcaUn9tmkjMRVNJhN9SQ7OeOO2P0fHtDQwYDkqu9AXipMF9I39ZI/frz74= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 - exec: client.0: - ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T10:19:24.969 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa; will attempt to use it 2024-09-16T10:19:24.970 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa/tasks 2024-09-16T10:19:24.970 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T10:19:24.972 INFO:teuthology.task.internal:Checking packages... 2024-09-16T10:19:24.991 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8' 2024-09-16T10:19:24.992 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T10:19:24.992 INFO:teuthology.packaging:ref: None 2024-09-16T10:19:24.992 INFO:teuthology.packaging:tag: None 2024-09-16T10:19:24.992 INFO:teuthology.packaging:branch: reef 2024-09-16T10:19:24.992 INFO:teuthology.packaging:sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:19:24.992 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-16T10:19:25.275 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-853.gae7f3733 2024-09-16T10:19:25.277 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T10:19:25.283 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T10:19:25.283 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T10:19:25.351 INFO:teuthology.task.internal:Saving configuration 2024-09-16T10:19:25.364 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T10:19:25.446 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T10:19:25.465 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 10:14:24.775275', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMEbhcnxotVjRvNXM4TQfLe90v6fM8oMlstd1MndhTfk6Ee+NUOdeja0vhjcKFi+2jKG7g1E5Z98W6dJ1yZWkGI='} 2024-09-16T10:19:25.485 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 10:14:24.777222', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjIz7uKVJ6cOLwfgH8Ei+hjNgGTTYcaUn9tmkjMRVNJhN9SQ7OeOO2P0fHtDQwYDkqu9AXipMF9I39ZI/frz74='} 2024-09-16T10:19:25.485 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T10:19:25.495 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T10:19:25.495 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T10:19:25.496 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T10:19:25.568 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fce1259bac0>, signals=[15]) 2024-09-16T10:19:25.568 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T10:19:25.584 INFO:teuthology.task.internal:Opening connections... 2024-09-16T10:19:25.584 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-09-16T10:19:25.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:19:25.662 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:19:25.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:19:25.738 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T10:19:25.749 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-09-16T10:19:25.767 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-09-16T10:19:25.768 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-09-16T10:19:25.825 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T10:19:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T10:19:25.827 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T10:19:25.827 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-09-16T10:19:25.853 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-09-16T10:19:25.872 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-09-16T10:19:25.872 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-09-16T10:19:25.930 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-09-16T10:19:25.930 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-09-16T10:19:25.930 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T10:19:25.931 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T10:19:25.932 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-09-16T10:19:25.953 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T10:19:25.962 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T10:19:25.982 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T10:19:25.982 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-09-16T10:19:25.985 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-09-16T10:19:26.002 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T10:19:26.038 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T10:19:26.039 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-09-16T10:19:26.042 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-09-16T10:19:26.075 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T10:19:26.126 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T10:19:26.147 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T10:19:26.148 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.f': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T10:19:26.148 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T10:19:26.148 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T10:19:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:26.165 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-09-16T10:19:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:26.166 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-09-16T10:19:26.224 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-503.el9.x86_64 2024-09-16T10:19:26.224 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-503.el9.x86_64 2024-09-16T10:19:26.224 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-09-16T10:19:26.225 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-503.el9.x86_64 2024-09-16T10:19:26.226 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-503.el9.x86_64 2024-09-16T10:19:26.226 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-09-16T10:19:28.667 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-16T10:19:28.695 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-16T10:19:33.604 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 5.4 MB/s | 20 MB 00:03 2024-09-16T10:19:33.859 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 20 MB 00:04 2024-09-16T10:19:37.997 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-16T10:19:38.360 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.5 MB 00:00 2024-09-16T10:19:39.497 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-16T10:19:39.877 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 19 kB 00:00 2024-09-16T10:19:40.876 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-16T10:19:40.982 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-16T10:19:46.404 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T10:19:46.466 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-09-16T10:19:48.427 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T10:19:48.427 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T10:19:48.480 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T10:19:48.480 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T10:19:48.527 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:19:48.533 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:19:48.533 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-09-16T10:19:48.533 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-09-16T10:19:48.534 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:19:48.535 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 79 M 2024-09-16T10:19:48.535 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 126 M 2024-09-16T10:19:48.535 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:19:48.567 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:19:48.573 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:19:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-16T10:19:48.573 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:19:48.573 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:19:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 79 M 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 126 M 2024-09-16T10:19:48.574 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:19:49.114 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 75 kB/s | 23 kB 00:00 2024-09-16T10:19:50.075 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 16 kB/s | 23 kB 00:01 2024-09-16T10:19:52.384 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.2 MB/s | 16 MB 00:03 2024-09-16T10:19:53.700 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.8 MB/s | 35 MB 00:05 2024-09-16T10:19:54.842 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.0 MB/s | 29 MB 00:04 2024-09-16T10:19:54.842 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:19:54.843 INFO:teuthology.orchestra.run.smithi094.stdout:Total 13 MB/s | 79 MB 00:06 2024-09-16T10:19:55.527 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:19:55.886 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:19:55.887 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:19:56.860 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:19:56.861 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:19:58.638 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 3.0 MB/s | 29 MB 00:09 2024-09-16T10:19:59.299 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:19:59.438 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 3.3 MB/s | 35 MB 00:10 2024-09-16T10:19:59.639 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 1.4 MB/s | 16 MB 00:10 2024-09-16T10:19:59.639 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:19:59.639 INFO:teuthology.orchestra.run.smithi152.stdout:Total 7.1 MB/s | 79 MB 00:11 2024-09-16T10:19:59.966 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T10:20:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T10:20:00.326 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:20:00.672 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:20:00.672 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:20:01.652 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:20:01.652 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:20:02.868 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T10:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T10:20:04.084 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:20:04.723 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T10:20:04.807 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T10:20:07.614 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T10:20:07.761 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T10:20:07.831 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T10:20:07.917 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:12.602 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T10:20:12.645 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:12.780 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:17.493 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:51.065 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:51.478 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:51.478 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T10:20:51.478 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T10:20:51.478 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T10:20:52.380 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:52.380 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:20:52.380 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:20:52.380 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T10:20:52.380 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T10:20:52.380 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T10:20:52.380 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T10:20:52.381 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:20:52.381 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:20:52.780 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T10:20:52.856 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T10:20:52.856 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T10:20:52.856 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T10:20:52.857 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T10:20:52.857 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T10:20:52.857 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T10:20:52.857 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T10:20:52.857 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T10:20:52.917 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T10:20:52.917 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T10:20:52.917 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2024-09-16T10:20:52.948 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2024-09-16T10:20:52.949 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2024-09-16T10:20:52.949 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.06 2024-09-16T10:20:52.949 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 82.el9 2024-09-16T10:20:52.949 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2024-09-16T10:20:52.949 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T10:20:52.949 INFO:teuthology.orchestra.run.smithi094.stdout:Group : Unspecified 2024-09-16T10:20:52.949 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 8274670 2024-09-16T10:20:52.949 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2024-09-16T10:20:52.949 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T10:20:52.950 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T10:20:52.950 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T10:20:52.950 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T10:20:52.950 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : builder@centos.org 2024-09-16T10:20:52.950 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2024-09-16T10:20:52.950 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T10:20:52.950 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2024-09-16T10:20:52.950 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2024-09-16T10:20:52.950 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:20:52.951 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T10:20:52.951 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T10:20:52.951 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T10:20:52.951 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2024-09-16T10:20:52.951 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:20:52.951 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T10:20:52.952 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T10:20:52.952 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 5.14.0-505.el9.x86_64 2024-09-16T10:20:52.952 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T10:20:53.685 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2024-09-16T10:20:55.036 INFO:teuthology.orchestra.run.smithi094.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T10:20:55.073 INFO:teuthology.orchestra.run.smithi094.stderr:done 2024-09-16T10:20:55.075 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-09-16T10:20:55.091 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.vr0LK7UNZ5 2024-09-16T10:20:55.091 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vr0LK7UNZ5 2024-09-16T10:20:55.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.vr0LK7UNZ5 2024-09-16T10:20:55.301 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.vr0LK7UNZ5 is 6KB 2024-09-16T10:20:55.349 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.vr0LK7UNZ5 2024-09-16T10:20:55.364 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T10:20:55.428 INFO:teuthology.orchestra.run.smithi094.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T10:20:55.428 INFO:teuthology.orchestra.run.smithi094.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T10:20:55.428 INFO:teuthology.orchestra.run.smithi094.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T10:20:55.428 INFO:teuthology.orchestra.run.smithi094.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T10:20:55.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T10:20:55.577 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2024-09-16T10:20:56.172 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:56.215 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:56.607 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:56.608 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T10:20:56.608 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T10:20:56.608 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T10:20:57.443 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:20:57.443 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:20:57.443 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:20:57.443 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T10:20:57.443 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T10:20:57.443 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T10:20:57.443 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T10:20:57.444 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:20:57.444 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:20:57.799 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T10:20:57.880 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T10:20:57.881 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T10:20:57.881 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T10:20:57.881 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T10:20:57.881 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T10:20:57.881 INFO:teuthology.task.kernel:Installing distro kernel on mon.f... 2024-09-16T10:20:57.881 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T10:20:57.881 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T10:20:57.941 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T10:20:57.944 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T10:20:57.944 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2024-09-16T10:20:57.975 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2024-09-16T10:20:57.975 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2024-09-16T10:20:57.975 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 82.el9 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8274670 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T10:20:57.976 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2024-09-16T10:20:57.977 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2024-09-16T10:20:57.977 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:20:57.977 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T10:20:57.977 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T10:20:57.977 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T10:20:57.977 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2024-09-16T10:20:57.977 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:20:57.977 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T10:20:57.978 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T10:20:57.978 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-505.el9.x86_64 2024-09-16T10:20:57.978 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T10:20:58.713 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-09-16T10:21:00.089 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T10:21:00.129 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-09-16T10:21:00.131 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-16T10:21:00.147 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.4kYWYNU2hy 2024-09-16T10:21:00.148 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4kYWYNU2hy 2024-09-16T10:21:00.214 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.4kYWYNU2hy 2024-09-16T10:21:00.354 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.4kYWYNU2hy is 6KB 2024-09-16T10:21:00.375 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.4kYWYNU2hy 2024-09-16T10:21:00.390 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T10:21:00.453 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T10:21:00.453 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T10:21:00.453 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T10:21:00.453 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T10:21:00.454 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T10:21:00.598 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-09-16T10:21:25.601 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T10:21:25.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-16T10:21:25.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:21:30.605 INFO:teuthology.task.kernel:Checking client mon.f for new kernel version... 2024-09-16T10:21:30.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T10:21:30.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:22:05.121 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-09-16T10:22:14.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T10:22:14.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:22:25.614 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T10:22:32.509 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-16T10:22:34.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:22:35.011 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-09-16T10:22:35.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-16T10:22:35.421 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T10:22:35.422 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-09-16T10:22:35.436 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-505.el9.x86_64 2024-09-16T10:22:35.436 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T10:22:35.437 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T10:22:35.437 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T10:22:36.438 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T10:22:36.438 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T10:22:36.521 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2024-09-16T10:22:36.551 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:44.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T10:22:44.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:22:44.892 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-09-16T10:22:45.381 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T10:22:45.381 INFO:teuthology.task.kernel:Checking kernel version of mon.f, want "5.14.0-505.el9.x86_64"... 2024-09-16T10:22:45.381 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-09-16T10:22:45.396 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-505.el9.x86_64 2024-09-16T10:22:45.396 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T10:22:45.397 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T10:22:45.397 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T10:22:46.397 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2024-09-16T10:22:46.398 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T10:22:46.478 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-09-16T10:22:46.502 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:46.502 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T10:22:46.512 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T10:22:46.512 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T10:22:46.515 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T10:22:46.531 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T10:22:46.593 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T10:22:46.678 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T10:22:46.678 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T10:22:46.681 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T10:22:46.809 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T10:22:46.815 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T10:22:46.815 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T10:22:46.818 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T10:22:46.846 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:22:46.848 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:22:46.857 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:22:46.858 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:22:46.860 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T10:22:46.867 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T10:22:46.867 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T10:22:46.902 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T10:22:46.928 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T10:22:46.937 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T10:22:46.937 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T10:22:46.971 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T10:22:46.986 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:22:47.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:22:47.128 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:22:47.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:22:47.252 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:22:47.252 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T10:22:47.318 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:22:47.361 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:22:47.405 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:22:47.487 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:22:47.527 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:22:47.527 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T10:22:47.601 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-09-16T10:22:47.604 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-09-16T10:22:47.653 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:22:47.685 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:22:48.156 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T10:22:48.163 INFO:teuthology.task.internal:Starting timer... 2024-09-16T10:22:48.163 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T10:22:48.183 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T10:22:48.196 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T10:22:48.196 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-09-16T10:22:48.271 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-09-16T10:22:48.273 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-09-16T10:22:48.330 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-09-16T10:22:48.332 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T10:22:48.332 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-09-16T10:22:48.383 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-09-16T10:22:48.383 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-09-16T10:22:48.412 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-09-16T10:22:48.412 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-09-16T10:22:48.412 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:22:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.441 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:22:48.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.475 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T10:22:48.476 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-09-16T10:22:48.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-09-16T10:22:48.556 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T10:22:48.566 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T10:22:48.578 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T10:22:48.579 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T10:26:32.062 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-09-16T10:26:32.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-16T10:26:32.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:26:32.143 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-09-16T10:26:32.224 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-16T10:26:32.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T10:26:32.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:26:32.309 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-09-16T10:26:32.384 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T10:26:32.384 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T10:26:32.395 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T10:26:32.395 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T10:26:32.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:26:32.398 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T10:26:32.399 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:26:32.424 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T10:26:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T10:26:32.453 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T10:26:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-09-16T10:26:32.468 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T10:26:32.473 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-09-16T10:26:32.487 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T10:26:32.492 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-09-16T10:26:32.501 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T10:26:32.503 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-09-16T10:26:32.519 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T10:26:32.532 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T10:26:32.560 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:26:32.562 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:26:32.563 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-09-16T10:26:32.563 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:26:32.563 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:26:32.563 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:26:32.563 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:26:32.591 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:26:32.595 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:26:32.595 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-09-16T10:26:32.595 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:26:32.595 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:26:32.595 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:26:32.595 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:26:32.596 INFO:teuthology.run_tasks:Running task install... 2024-09-16T10:26:32.606 DEBUG:teuthology.task.install:project ceph 2024-09-16T10:26:32.606 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}} 2024-09-16T10:26:32.606 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'} 2024-09-16T10:26:32.607 INFO:teuthology.task.install:Using flavor: default 2024-09-16T10:26:32.612 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T10:26:32.612 INFO:teuthology.task.install:extra packages: [] 2024-09-16T10:26:32.613 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'tag': None, 'wait_for_package': False} 2024-09-16T10:26:32.614 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:26:32.616 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'tag': None, 'wait_for_package': False} 2024-09-16T10:26:32.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:26:32.761 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/ 2024-09-16T10:26:32.761 INFO:teuthology.task.install.rpm:Package version is 18.2.4-845.ge4e227b3 2024-09-16T10:26:32.767 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/ 2024-09-16T10:26:32.767 INFO:teuthology.task.install.rpm:Package version is 18.2.4-845.ge4e227b3 2024-09-16T10:26:32.872 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T10:26:32.872 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:26:32.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T10:26:32.878 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T10:26:32.878 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:26:32.878 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T10:26:32.899 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T10:26:32.899 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T10:26:32.903 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T10:26:32.903 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T10:26:32.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T10:26:32.969 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T10:26:33.040 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T10:26:33.044 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T10:26:33.104 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-09-16T10:26:33.105 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-09-16T10:26:33.109 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-09-16T10:26:33.110 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-09-16T10:26:33.329 INFO:teuthology.orchestra.run.smithi094.stdout:45 files removed 2024-09-16T10:26:33.345 INFO:teuthology.orchestra.run.smithi152.stdout:45 files removed 2024-09-16T10:26:33.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2024-09-16T10:26:33.372 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2024-09-16T10:26:33.898 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 300 kB/s | 77 kB 00:00 2024-09-16T10:26:33.940 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 277 kB/s | 77 kB 00:00 2024-09-16T10:26:34.174 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 50 kB/s | 11 kB 00:00 2024-09-16T10:26:34.216 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 50 kB/s | 11 kB 00:00 2024-09-16T10:26:34.432 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 8.4 kB/s | 1.9 kB 00:00 2024-09-16T10:26:34.475 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 8.4 kB/s | 1.9 kB 00:00 2024-09-16T10:26:35.475 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 8.5 MB/s | 8.2 MB 00:00 2024-09-16T10:26:36.133 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.2 MB 00:01 2024-09-16T10:26:40.484 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 20 MB 00:03 2024-09-16T10:26:40.793 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 20 MB 00:03 2024-09-16T10:26:45.943 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-16T10:26:46.169 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.5 MB 00:01 2024-09-16T10:26:47.602 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 19 kB 00:00 2024-09-16T10:26:47.886 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 19 kB 00:00 2024-09-16T10:26:48.489 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-16T10:26:48.707 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-16T10:26:53.995 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T10:26:54.172 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-09-16T10:26:56.362 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:26:56.382 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:26:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:26:56.382 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:26:56.382 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:26:56.382 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 7.7 M 2024-09-16T10:26:56.383 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-09-16T10:26:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 3.3 M 2024-09-16T10:26:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 3.0 M 2024-09-16T10:26:56.383 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:26:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T10:26:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 5.1 M 2024-09-16T10:26:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 18 M 2024-09-16T10:26:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 25 k 2024-09-16T10:26:56.383 INFO:teuthology.orchestra.run.smithi152.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 712 k 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 478 k 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 4.5 M 2024-09-16T10:26:56.384 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 45 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 129 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 162 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 321 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 300 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 99 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T10:26:56.385 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T10:26:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:26:56.386 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:26:56.386 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:26:56.386 INFO:teuthology.orchestra.run.smithi152.stdout:Install 28 Packages 2024-09-16T10:26:56.386 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-09-16T10:26:56.386 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:26:56.387 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 53 M 2024-09-16T10:26:56.387 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:26:56.554 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:26:56.563 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:26:56.563 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:26:56.563 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:26:56.563 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 7.7 M 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 3.3 M 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 3.0 M 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 5.1 M 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 18 M 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 25 k 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T10:26:56.564 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 712 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 478 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 4.5 M 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 45 k 2024-09-16T10:26:56.565 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 129 k 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 162 k 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 321 k 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 300 k 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 99 k 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout:Install 28 Packages 2024-09-16T10:26:56.566 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-09-16T10:26:56.567 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:26:56.567 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 53 M 2024-09-16T10:26:56.567 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:26:57.213 INFO:teuthology.orchestra.run.smithi152.stdout:(1/30): ceph-radosgw-18.2.4-845.ge4e227b3.el9.x 11 MB/s | 7.7 MB 00:00 2024-09-16T10:26:57.280 INFO:teuthology.orchestra.run.smithi152.stdout:(2/30): ceph-base-18.2.4-845.ge4e227b3.el9.x86_ 6.7 MB/s | 5.1 MB 00:00 2024-09-16T10:26:57.305 INFO:teuthology.orchestra.run.smithi152.stdout:(3/30): ceph-selinux-18.2.4-845.ge4e227b3.el9.x 274 kB/s | 25 kB 00:00 2024-09-16T10:26:57.389 INFO:teuthology.orchestra.run.smithi152.stdout:(4/30): libcephfs2-18.2.4-845.ge4e227b3.el9.x86 6.4 MB/s | 712 kB 00:00 2024-09-16T10:26:57.565 INFO:teuthology.orchestra.run.smithi094.stdout:(1/30): ceph-base-18.2.4-845.ge4e227b3.el9.x86_ 6.7 MB/s | 5.1 MB 00:00 2024-09-16T10:26:57.623 INFO:teuthology.orchestra.run.smithi094.stdout:(2/30): ceph-selinux-18.2.4-845.ge4e227b3.el9.x 433 kB/s | 25 kB 00:00 2024-09-16T10:26:57.706 INFO:teuthology.orchestra.run.smithi152.stdout:(5/30): ceph-common-18.2.4-845.ge4e227b3.el9.x8 15 MB/s | 18 MB 00:01 2024-09-16T10:26:57.740 INFO:teuthology.orchestra.run.smithi094.stdout:(3/30): libcephfs2-18.2.4-845.ge4e227b3.el9.x86 6.0 MB/s | 712 kB 00:00 2024-09-16T10:26:57.789 INFO:teuthology.orchestra.run.smithi152.stdout:(6/30): librgw2-18.2.4-845.ge4e227b3.el9.x86_64 11 MB/s | 4.5 MB 00:00 2024-09-16T10:26:57.815 INFO:teuthology.orchestra.run.smithi152.stdout:(7/30): python3-ceph-argparse-18.2.4-845.ge4e22 415 kB/s | 45 kB 00:00 2024-09-16T10:26:57.856 INFO:teuthology.orchestra.run.smithi152.stdout:(8/30): python3-ceph-common-18.2.4-845.ge4e227b 1.9 MB/s | 129 kB 00:00 2024-09-16T10:26:57.865 INFO:teuthology.orchestra.run.smithi094.stdout:(4/30): libradosstriper1-18.2.4-845.ge4e227b3.e 3.7 MB/s | 478 kB 00:00 2024-09-16T10:26:57.882 INFO:teuthology.orchestra.run.smithi152.stdout:(9/30): python3-cephfs-18.2.4-845.ge4e227b3.el9 2.4 MB/s | 162 kB 00:00 2024-09-16T10:26:57.907 INFO:teuthology.orchestra.run.smithi152.stdout:(10/30): python3-rados-18.2.4-845.ge4e227b3.el9 6.2 MB/s | 321 kB 00:00 2024-09-16T10:26:57.966 INFO:teuthology.orchestra.run.smithi152.stdout:(11/30): python3-rbd-18.2.4-845.ge4e227b3.el9.x 3.5 MB/s | 300 kB 00:00 2024-09-16T10:26:57.991 INFO:teuthology.orchestra.run.smithi152.stdout:(12/30): python3-rgw-18.2.4-845.ge4e227b3.el9.x 1.2 MB/s | 99 kB 00:00 2024-09-16T10:26:58.008 INFO:teuthology.orchestra.run.smithi094.stdout:(5/30): ceph-radosgw-18.2.4-845.ge4e227b3.el9.x 6.4 MB/s | 7.7 MB 00:01 2024-09-16T10:26:58.033 INFO:teuthology.orchestra.run.smithi152.stdout:(13/30): libradosstriper1-18.2.4-845.ge4e227b3. 657 kB/s | 478 kB 00:00 2024-09-16T10:26:58.092 INFO:teuthology.orchestra.run.smithi152.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 853 kB/s | 107 kB 00:00 2024-09-16T10:26:58.133 INFO:teuthology.orchestra.run.smithi152.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-09-16T10:26:58.150 INFO:teuthology.orchestra.run.smithi152.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 386 kB/s | 45 kB 00:00 2024-09-16T10:26:58.191 INFO:teuthology.orchestra.run.smithi094.stdout:(6/30): ceph-common-18.2.4-845.ge4e227b3.el9.x8 13 MB/s | 18 MB 00:01 2024-09-16T10:26:58.192 INFO:teuthology.orchestra.run.smithi152.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-16T10:26:58.216 INFO:teuthology.orchestra.run.smithi094.stdout:(7/30): python3-ceph-argparse-18.2.4-845.ge4e22 216 kB/s | 45 kB 00:00 2024-09-16T10:26:58.234 INFO:teuthology.orchestra.run.smithi152.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 505 kB/s | 42 kB 00:00 2024-09-16T10:26:58.242 INFO:teuthology.orchestra.run.smithi094.stdout:(8/30): python3-ceph-common-18.2.4-845.ge4e227b 2.5 MB/s | 129 kB 00:00 2024-09-16T10:26:58.259 INFO:teuthology.orchestra.run.smithi152.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-16T10:26:58.275 INFO:teuthology.orchestra.run.smithi094.stdout:(9/30): python3-cephfs-18.2.4-845.ge4e227b3.el9 2.7 MB/s | 162 kB 00:00 2024-09-16T10:26:58.293 INFO:teuthology.orchestra.run.smithi152.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 1.1 MB/s | 111 kB 00:00 2024-09-16T10:26:58.309 INFO:teuthology.orchestra.run.smithi094.stdout:(10/30): python3-rados-18.2.4-845.ge4e227b3.el9 4.7 MB/s | 321 kB 00:00 2024-09-16T10:26:58.318 INFO:teuthology.orchestra.run.smithi152.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.6 MB/s | 26 kB 00:00 2024-09-16T10:26:58.334 INFO:teuthology.orchestra.run.smithi094.stdout:(11/30): python3-rbd-18.2.4-845.ge4e227b3.el9.x 5.0 MB/s | 300 kB 00:00 2024-09-16T10:26:58.344 INFO:teuthology.orchestra.run.smithi152.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-16T10:26:58.360 INFO:teuthology.orchestra.run.smithi094.stdout:(12/30): python3-rgw-18.2.4-845.ge4e227b3.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-16T10:26:58.377 INFO:teuthology.orchestra.run.smithi152.stdout:(23/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.1 MB/s | 308 kB 00:00 2024-09-16T10:26:58.402 INFO:teuthology.orchestra.run.smithi152.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-16T10:26:58.435 INFO:teuthology.orchestra.run.smithi094.stdout:(13/30): librgw2-18.2.4-845.ge4e227b3.el9.x86_6 7.8 MB/s | 4.5 MB 00:00 2024-09-16T10:26:58.444 INFO:teuthology.orchestra.run.smithi152.stdout:(25/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-16T10:26:58.478 INFO:teuthology.orchestra.run.smithi152.stdout:(26/30): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi094.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 581 kB/s | 107 kB 00:00 2024-09-16T10:26:58.544 INFO:teuthology.orchestra.run.smithi094.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 416 kB/s | 45 kB 00:00 2024-09-16T10:26:58.553 INFO:teuthology.orchestra.run.smithi152.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-16T10:26:58.569 INFO:teuthology.orchestra.run.smithi094.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 765 kB/s | 160 kB 00:00 2024-09-16T10:26:58.611 INFO:teuthology.orchestra.run.smithi094.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-16T10:26:58.637 INFO:teuthology.orchestra.run.smithi152.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-16T10:26:58.645 INFO:teuthology.orchestra.run.smithi094.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-09-16T10:26:58.670 INFO:teuthology.orchestra.run.smithi094.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-16T10:26:58.712 INFO:teuthology.orchestra.run.smithi152.stdout:(29/30): librados2-18.2.4-845.ge4e227b3.el9.x86 14 MB/s | 3.3 MB 00:00 2024-09-16T10:26:58.721 INFO:teuthology.orchestra.run.smithi094.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 1.0 MB/s | 111 kB 00:00 2024-09-16T10:26:58.746 INFO:teuthology.orchestra.run.smithi094.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-16T10:26:58.771 INFO:teuthology.orchestra.run.smithi094.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T10:26:58.779 INFO:teuthology.orchestra.run.smithi152.stdout:(30/30): librbd1-18.2.4-845.ge4e227b3.el9.x86_6 13 MB/s | 3.0 MB 00:00 2024-09-16T10:26:58.783 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:26:58.783 INFO:teuthology.orchestra.run.smithi152.stdout:Total 22 MB/s | 53 MB 00:02 2024-09-16T10:26:58.805 INFO:teuthology.orchestra.run.smithi094.stdout:(23/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.9 MB/s | 308 kB 00:00 2024-09-16T10:26:58.830 INFO:teuthology.orchestra.run.smithi094.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-16T10:26:58.854 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:26:58.864 INFO:teuthology.orchestra.run.smithi094.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T10:26:58.889 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:26:58.889 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:26:58.914 INFO:teuthology.orchestra.run.smithi094.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-16T10:26:58.973 INFO:teuthology.orchestra.run.smithi094.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-16T10:26:59.057 INFO:teuthology.orchestra.run.smithi094.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-16T10:26:59.182 INFO:teuthology.orchestra.run.smithi094.stdout:(29/30): librbd1-18.2.4-845.ge4e227b3.el9.x86_6 14 MB/s | 3.0 MB 00:00 2024-09-16T10:26:59.249 INFO:teuthology.orchestra.run.smithi094.stdout:(30/30): librados2-18.2.4-845.ge4e227b3.el9.x86 9.9 MB/s | 3.3 MB 00:00 2024-09-16T10:26:59.254 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:26:59.254 INFO:teuthology.orchestra.run.smithi094.stdout:Total 20 MB/s | 53 MB 00:02 2024-09-16T10:26:59.330 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:26:59.366 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:26:59.367 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:26:59.530 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:26:59.530 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:26:59.988 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:26:59.988 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:27:00.921 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:27:01.013 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-16T10:27:01.042 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-16T10:27:01.092 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-16T10:27:01.381 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:27:01.462 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-16T10:27:01.463 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-16T10:27:01.495 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-16T10:27:01.516 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 5/32 2024-09-16T10:27:01.545 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-16T10:27:01.608 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 5/32 2024-09-16T10:27:01.691 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 6/32 2024-09-16T10:27:01.717 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 7/32 2024-09-16T10:27:01.780 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 7/32 2024-09-16T10:27:01.808 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-16T10:27:01.860 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:18.2.4-845.ge4e227b3.el9.x 9/32 2024-09-16T10:27:01.910 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-16T10:27:01.935 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 10/32 2024-09-16T10:27:01.961 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 11/32 2024-09-16T10:27:01.969 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 5/32 2024-09-16T10:27:02.035 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 11/32 2024-09-16T10:27:02.045 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 5/32 2024-09-16T10:27:02.129 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 6/32 2024-09-16T10:27:02.141 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-16T10:27:02.154 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 7/32 2024-09-16T10:27:02.177 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-16T10:27:02.217 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 7/32 2024-09-16T10:27:02.222 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-16T10:27:02.246 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-16T10:27:02.250 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-16T10:27:02.277 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-16T10:27:02.292 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:18.2.4-845.ge4e227b3.el9.x 9/32 2024-09-16T10:27:02.313 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-16T10:27:02.346 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-16T10:27:02.364 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 10/32 2024-09-16T10:27:02.398 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 11/32 2024-09-16T10:27:02.465 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 11/32 2024-09-16T10:27:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-16T10:27:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-16T10:27:02.646 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-16T10:27:02.658 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-16T10:27:02.688 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-16T10:27:02.691 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 20/32 2024-09-16T10:27:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-16T10:27:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-16T10:27:02.782 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 20/32 2024-09-16T10:27:02.783 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-16T10:27:02.821 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 21/32 2024-09-16T10:27:02.866 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-16T10:27:03.091 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-16T10:27:03.138 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 20/32 2024-09-16T10:27:03.212 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 20/32 2024-09-16T10:27:03.243 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 21/32 2024-09-16T10:27:03.270 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-16T10:27:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-16T10:27:03.302 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 24/32 2024-09-16T10:27:03.356 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 24/32 2024-09-16T10:27:03.425 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 25/32 2024-09-16T10:27:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:18.2.4-845.ge4e227b3.el9.x86 26/32 2024-09-16T10:27:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-16T10:27:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 24/32 2024-09-16T10:27:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 24/32 2024-09-16T10:27:03.863 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 25/32 2024-09-16T10:27:03.901 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:18.2.4-845.ge4e227b3.el9.x86 26/32 2024-09-16T10:27:05.559 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 27/32 2024-09-16T10:27:05.600 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 27/32 2024-09-16T10:27:06.003 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 27/32 2024-09-16T10:27:06.040 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 27/32 2024-09-16T10:27:06.169 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 27/32 2024-09-16T10:27:06.212 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 28/32 2024-09-16T10:27:06.318 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 28/32 2024-09-16T10:27:06.318 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T10:27:06.318 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T10:27:06.318 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:06.346 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 29/32 2024-09-16T10:27:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 27/32 2024-09-16T10:27:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 28/32 2024-09-16T10:27:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 28/32 2024-09-16T10:27:06.728 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T10:27:06.728 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T10:27:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 29/32 2024-09-16T10:27:16.330 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 29/32 2024-09-16T10:27:16.330 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-09-16T10:27:16.330 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-09-16T10:27:16.330 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-09-16T10:27:16.330 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-09-16T10:27:16.330 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-09-16T10:27:16.330 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-09-16T10:27:16.331 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-09-16T10:27:16.331 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 29/32 2024-09-16T10:27:16.917 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-09-16T10:27:16.917 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-09-16T10:27:16.918 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-09-16T10:27:16.918 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-09-16T10:27:16.918 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-09-16T10:27:16.918 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-09-16T10:27:16.918 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-09-16T10:27:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:17.075 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 30/32 2024-09-16T10:27:17.100 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 30/32 2024-09-16T10:27:17.100 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:17.100 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T10:27:17.100 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:27:17.101 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:27:17.101 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:17.102 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-16T10:27:17.185 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-16T10:27:17.185 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-16T10:27:17.684 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 30/32 2024-09-16T10:27:17.709 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 30/32 2024-09-16T10:27:17.709 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:17.709 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T10:27:17.709 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:27:17.709 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:27:17.709 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:17.711 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-16T10:27:17.794 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-16T10:27:17.795 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-16T10:27:18.300 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 1/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 2/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 3/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 4/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 5/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 6/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 7/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:18.2.4-845.ge4e227b3.el9.x 8/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:18.2.4-845.ge4e227b3.el9.x86 9/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 10/32 2024-09-16T10:27:18.301 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 11/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 12/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 13/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-16T10:27:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-16T10:27:18.303 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-16T10:27:18.303 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-16T10:27:18.303 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-16T10:27:18.303 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-16T10:27:18.303 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-16T10:27:18.303 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 29/32 2024-09-16T10:27:18.303 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-16T10:27:18.303 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 31/32 2024-09-16T10:27:18.878 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 1/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 2/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 3/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 4/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 5/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 6/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 7/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:18.2.4-845.ge4e227b3.el9.x 8/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:18.2.4-845.ge4e227b3.el9.x86 9/32 2024-09-16T10:27:18.879 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 10/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 11/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 12/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 13/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-16T10:27:18.880 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 29/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-16T10:27:18.881 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 31/32 2024-09-16T10:27:19.190 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-16T10:27:19.190 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:19.190 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-09-16T10:27:19.190 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T10:27:19.191 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T10:27:19.192 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.193 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.193 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.193 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T10:27:19.193 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T10:27:19.193 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:19.193 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:27:19.570 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2024-09-16T10:27:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-16T10:27:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:19.955 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-09-16T10:27:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.955 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:27:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T10:27:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T10:27:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T10:27:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:19.958 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:27:20.142 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:27 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:27:20.273 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:27:20.273 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:20.273 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 40 M 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 166 k 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-09-16T10:27:20.274 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:20.275 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 40 M 2024-09-16T10:27:20.275 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 164 M 2024-09-16T10:27:20.275 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:27:20.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2024-09-16T10:27:20.547 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-16T10:27:20.622 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): libcephsqlite-18.2.4-845.ge4e227b3.el9.x 800 kB/s | 166 kB 00:00 2024-09-16T10:27:20.639 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 696 kB/s | 64 kB 00:00 2024-09-16T10:27:20.903 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:26 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:27:21.035 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:27:21.035 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:27:21.035 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:21.035 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 40 M 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 166 k 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-09-16T10:27:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:21.037 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 40 M 2024-09-16T10:27:21.037 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 164 M 2024-09-16T10:27:21.037 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:27:21.336 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-16T10:27:21.386 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-16T10:27:21.411 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): libcephsqlite-18.2.4-845.ge4e227b3.el9.x 508 kB/s | 166 kB 00:00 2024-09-16T10:27:21.714 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): ceph-test-18.2.4-845.ge4e227b3.el9.x86_6 30 MB/s | 40 MB 00:01 2024-09-16T10:27:21.716 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:21.717 INFO:teuthology.orchestra.run.smithi152.stdout:Total 28 MB/s | 40 MB 00:01 2024-09-16T10:27:21.745 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:27:21.765 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:27:21.765 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:27:22.095 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:27:22.095 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:27:22.445 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): ceph-test-18.2.4-845.ge4e227b3.el9.x86_6 29 MB/s | 40 MB 00:01 2024-09-16T10:27:22.447 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:22.447 INFO:teuthology.orchestra.run.smithi094.stdout:Total 28 MB/s | 40 MB 00:01 2024-09-16T10:27:22.475 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:27:22.494 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:27:22.494 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:27:22.588 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:27:22.666 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T10:27:22.705 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T10:27:22.733 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 3/4 2024-09-16T10:27:22.822 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:27:22.823 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:27:23.292 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:27:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T10:27:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T10:27:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 3/4 2024-09-16T10:27:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 3/4 2024-09-16T10:27:26.513 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 4/4 2024-09-16T10:27:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 3/4 2024-09-16T10:27:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 4/4 2024-09-16T10:27:28.530 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 4/4 2024-09-16T10:27:28.530 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 1/4 2024-09-16T10:27:28.531 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 2/4 2024-09-16T10:27:28.531 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T10:27:28.826 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T10:27:28.826 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:28.827 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:27:28.827 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:28.827 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:28.827 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T10:27:28.827 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T10:27:28.827 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:28.827 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:27:29.134 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2024-09-16T10:27:29.280 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 4/4 2024-09-16T10:27:29.281 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 1/4 2024-09-16T10:27:29.281 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 2/4 2024-09-16T10:27:29.281 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T10:27:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T10:27:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:29.593 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:27:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T10:27:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T10:27:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:29.594 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:27:29.701 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:36 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:27:29.845 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:27:29.847 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:29.847 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:27:29.847 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:29.847 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 6.5 k 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 2.1 M 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 1.5 M 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 246 k 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 4.7 M 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 17 M 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T10:27:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T10:27:29.849 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T10:27:29.850 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T10:27:29.850 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T10:27:29.850 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T10:27:29.850 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T10:27:29.850 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T10:27:29.850 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T10:27:29.850 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T10:27:29.850 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T10:27:29.850 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T10:27:29.851 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T10:27:29.851 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T10:27:29.851 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T10:27:29.851 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:29.851 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:27:29.851 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:29.851 INFO:teuthology.orchestra.run.smithi152.stdout:Install 33 Packages 2024-09-16T10:27:29.851 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:29.852 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 29 M 2024-09-16T10:27:29.852 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 104 M 2024-09-16T10:27:29.852 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:27:29.927 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2024-09-16T10:27:30.055 INFO:teuthology.orchestra.run.smithi152.stdout:(1/33): ceph-18.2.4-845.ge4e227b3.el9.x86_64.rp 50 kB/s | 6.5 kB 00:00 2024-09-16T10:27:30.231 INFO:teuthology.orchestra.run.smithi152.stdout:(2/33): ceph-mgr-18.2.4-845.ge4e227b3.el9.x86_6 4.8 MB/s | 1.5 MB 00:00 2024-09-16T10:27:30.281 INFO:teuthology.orchestra.run.smithi152.stdout:(3/33): ceph-mds-18.2.4-845.ge4e227b3.el9.x86_6 5.9 MB/s | 2.1 MB 00:00 2024-09-16T10:27:30.373 INFO:teuthology.orchestra.run.smithi152.stdout:(4/33): ceph-mgr-modules-core-18.2.4-845.ge4e22 2.6 MB/s | 246 kB 00:00 2024-09-16T10:27:30.440 INFO:teuthology.orchestra.run.smithi152.stdout:(5/33): ceph-mon-18.2.4-845.ge4e227b3.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-16T10:27:30.495 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:36 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:27:30.557 INFO:teuthology.orchestra.run.smithi152.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-16T10:27:30.598 INFO:teuthology.orchestra.run.smithi152.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T10:27:30.631 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:27:30.633 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:30.633 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 6.5 k 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 2.1 M 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 1.5 M 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 246 k 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 4.7 M 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 17 M 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T10:27:30.634 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T10:27:30.635 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T10:27:30.636 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T10:27:30.637 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:30.637 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:27:30.637 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:30.637 INFO:teuthology.orchestra.run.smithi094.stdout:Install 33 Packages 2024-09-16T10:27:30.637 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:30.637 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 29 M 2024-09-16T10:27:30.637 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 104 M 2024-09-16T10:27:30.638 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:27:30.641 INFO:teuthology.orchestra.run.smithi152.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 6.1 MB/s | 1.2 MB 00:00 2024-09-16T10:27:30.666 INFO:teuthology.orchestra.run.smithi152.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-16T10:27:30.691 INFO:teuthology.orchestra.run.smithi152.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-16T10:27:30.716 INFO:teuthology.orchestra.run.smithi152.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-16T10:27:30.900 INFO:teuthology.orchestra.run.smithi152.stdout:(12/33): ceph-osd-18.2.4-845.ge4e227b3.el9.x86_ 25 MB/s | 17 MB 00:00 2024-09-16T10:27:31.307 INFO:teuthology.orchestra.run.smithi094.stdout:(1/33): ceph-18.2.4-845.ge4e227b3.el9.x86_64.rp 53 kB/s | 6.5 kB 00:00 2024-09-16T10:27:31.384 INFO:teuthology.orchestra.run.smithi152.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 52 kB/s | 35 kB 00:00 2024-09-16T10:27:31.474 INFO:teuthology.orchestra.run.smithi094.stdout:(2/33): ceph-mgr-18.2.4-845.ge4e227b3.el9.x86_6 5.0 MB/s | 1.5 MB 00:00 2024-09-16T10:27:31.483 INFO:teuthology.orchestra.run.smithi152.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 87 kB/s | 51 kB 00:00 2024-09-16T10:27:31.524 INFO:teuthology.orchestra.run.smithi094.stdout:(3/33): ceph-mds-18.2.4-845.ge4e227b3.el9.x86_6 6.1 MB/s | 2.1 MB 00:00 2024-09-16T10:27:31.551 INFO:teuthology.orchestra.run.smithi152.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 656 kB/s | 43 kB 00:00 2024-09-16T10:27:31.576 INFO:teuthology.orchestra.run.smithi152.stdout:(16/33): python3-mako-1.1.4-6.el9.noarch.rpm 195 kB/s | 172 kB 00:00 2024-09-16T10:27:31.609 INFO:teuthology.orchestra.run.smithi152.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-16T10:27:31.616 INFO:teuthology.orchestra.run.smithi094.stdout:(4/33): ceph-mgr-modules-core-18.2.4-845.ge4e22 2.6 MB/s | 246 kB 00:00 2024-09-16T10:27:31.635 INFO:teuthology.orchestra.run.smithi152.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 462 kB/s | 11 kB 00:00 2024-09-16T10:27:31.660 INFO:teuthology.orchestra.run.smithi152.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-16T10:27:31.685 INFO:teuthology.orchestra.run.smithi152.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 930 kB/s | 23 kB 00:00 2024-09-16T10:27:31.692 INFO:teuthology.orchestra.run.smithi094.stdout:(5/33): ceph-mon-18.2.4-845.ge4e227b3.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-16T10:27:31.728 INFO:teuthology.orchestra.run.smithi152.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-09-16T10:27:31.753 INFO:teuthology.orchestra.run.smithi152.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 289 kB/s | 19 kB 00:00 2024-09-16T10:27:31.778 INFO:teuthology.orchestra.run.smithi152.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 400 kB/s | 20 kB 00:00 2024-09-16T10:27:31.803 INFO:teuthology.orchestra.run.smithi152.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-09-16T10:27:31.828 INFO:teuthology.orchestra.run.smithi152.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T10:27:31.854 INFO:teuthology.orchestra.run.smithi152.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-16T10:27:31.879 INFO:teuthology.orchestra.run.smithi152.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-16T10:27:31.904 INFO:teuthology.orchestra.run.smithi152.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-16T10:27:31.929 INFO:teuthology.orchestra.run.smithi152.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-16T10:27:31.954 INFO:teuthology.orchestra.run.smithi152.stdout:(30/33): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-16T10:27:31.980 INFO:teuthology.orchestra.run.smithi152.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 801 kB/s | 20 kB 00:00 2024-09-16T10:27:32.005 INFO:teuthology.orchestra.run.smithi152.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-16T10:27:32.033 INFO:teuthology.orchestra.run.smithi152.stdout:(33/33): python3-toml-0.10.2-6.el9.noarch.rpm 64 kB/s | 42 kB 00:00 2024-09-16T10:27:32.040 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:32.040 INFO:teuthology.orchestra.run.smithi152.stdout:Total 13 MB/s | 29 MB 00:02 2024-09-16T10:27:32.042 INFO:teuthology.orchestra.run.smithi094.stdout:(6/33): ceph-osd-18.2.4-845.ge4e227b3.el9.x86_6 29 MB/s | 17 MB 00:00 2024-09-16T10:27:32.161 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:27:32.198 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:27:32.198 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:27:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 400 kB/s | 253 kB 00:00 2024-09-16T10:27:32.371 INFO:teuthology.orchestra.run.smithi094.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-16T10:27:32.480 INFO:teuthology.orchestra.run.smithi094.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 1.1 MB/s | 125 kB 00:00 2024-09-16T10:27:32.505 INFO:teuthology.orchestra.run.smithi094.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 230 kB/s | 106 kB 00:00 2024-09-16T10:27:32.597 INFO:teuthology.orchestra.run.smithi094.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-09-16T10:27:32.628 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:27:32.629 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:27:32.689 INFO:teuthology.orchestra.run.smithi094.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:00 2024-09-16T10:27:32.715 INFO:teuthology.orchestra.run.smithi094.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 823 kB/s | 172 kB 00:00 2024-09-16T10:27:32.732 INFO:teuthology.orchestra.run.smithi094.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 259 kB/s | 35 kB 00:00 2024-09-16T10:27:32.757 INFO:teuthology.orchestra.run.smithi094.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 996 kB/s | 42 kB 00:00 2024-09-16T10:27:32.782 INFO:teuthology.orchestra.run.smithi094.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 867 kB/s | 43 kB 00:00 2024-09-16T10:27:32.832 INFO:teuthology.orchestra.run.smithi094.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.3 MB/s | 172 kB 00:00 2024-09-16T10:27:32.849 INFO:teuthology.orchestra.run.smithi094.stdout:(18/33): python3-pytz-2021.1-5.el9.noarch.rpm 319 kB/s | 51 kB 00:00 2024-09-16T10:27:32.866 INFO:teuthology.orchestra.run.smithi094.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 318 kB/s | 11 kB 00:00 2024-09-16T10:27:32.908 INFO:teuthology.orchestra.run.smithi094.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-16T10:27:32.933 INFO:teuthology.orchestra.run.smithi094.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 386 kB/s | 23 kB 00:00 2024-09-16T10:27:32.984 INFO:teuthology.orchestra.run.smithi094.stdout:(22/33): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-09-16T10:27:33.009 INFO:teuthology.orchestra.run.smithi094.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 197 kB/s | 19 kB 00:00 2024-09-16T10:27:33.034 INFO:teuthology.orchestra.run.smithi094.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 195 kB/s | 20 kB 00:00 2024-09-16T10:27:33.060 INFO:teuthology.orchestra.run.smithi094.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 928 kB/s | 46 kB 00:00 2024-09-16T10:27:33.085 INFO:teuthology.orchestra.run.smithi094.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-16T10:27:33.110 INFO:teuthology.orchestra.run.smithi094.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-16T10:27:33.135 INFO:teuthology.orchestra.run.smithi094.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-16T10:27:33.161 INFO:teuthology.orchestra.run.smithi094.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-16T10:27:33.186 INFO:teuthology.orchestra.run.smithi094.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-16T10:27:33.220 INFO:teuthology.orchestra.run.smithi094.stdout:(31/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-16T10:27:33.245 INFO:teuthology.orchestra.run.smithi094.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-09-16T10:27:33.270 INFO:teuthology.orchestra.run.smithi094.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T10:27:33.274 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:33.274 INFO:teuthology.orchestra.run.smithi094.stdout:Total 11 MB/s | 29 MB 00:02 2024-09-16T10:27:33.394 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:27:33.431 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:27:33.431 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:27:33.850 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:27:33.851 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:27:33.851 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:27:33.911 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-16T10:27:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-16T10:27:33.976 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-16T10:27:34.071 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-16T10:27:34.148 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-16T10:27:34.196 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-16T10:27:34.240 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-16T10:27:34.269 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-16T10:27:34.303 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-16T10:27:34.337 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-16T10:27:34.377 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-16T10:27:34.421 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-16T10:27:34.455 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-16T10:27:34.485 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-16T10:27:34.560 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-16T10:27:34.724 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-16T10:27:34.819 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-16T10:27:34.871 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-16T10:27:34.909 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-16T10:27:34.967 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-16T10:27:35.023 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-16T10:27:35.089 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:27:35.149 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-16T10:27:35.189 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-16T10:27:35.214 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-16T10:27:35.230 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-16T10:27:35.271 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-16T10:27:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-16T10:27:35.351 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-16T10:27:35.416 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-16T10:27:35.457 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-16T10:27:35.457 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-16T10:27:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-16T10:27:35.520 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-16T10:27:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-16T10:27:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-16T10:27:35.599 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-16T10:27:35.640 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-16T10:27:35.640 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-16T10:27:35.683 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-16T10:27:35.726 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-16T10:27:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-16T10:27:35.796 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-845.ge4e227b3.el9.n 28/33 2024-09-16T10:27:35.822 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 29/33 2024-09-16T10:27:35.837 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-16T10:27:35.847 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 29/33 2024-09-16T10:27:35.847 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:35.847 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T10:27:35.848 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:27:35.848 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:27:35.848 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:35.974 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-16T10:27:36.060 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-16T10:27:36.122 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-16T10:27:36.161 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-16T10:27:36.211 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-16T10:27:36.266 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-16T10:27:36.469 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-16T10:27:36.524 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-16T10:27:36.594 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-16T10:27:36.699 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-16T10:27:36.740 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-16T10:27:36.855 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-16T10:27:37.007 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-845.ge4e227b3.el9.n 28/33 2024-09-16T10:27:37.034 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 29/33 2024-09-16T10:27:37.059 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 29/33 2024-09-16T10:27:37.060 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:37.060 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T10:27:37.060 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:27:37.060 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:27:37.060 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:37.317 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 30/33 2024-09-16T10:27:37.346 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 30/33 2024-09-16T10:27:37.346 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:37.346 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T10:27:37.346 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:27:37.346 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:27:37.346 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:37.802 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 31/33 2024-09-16T10:27:37.825 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 31/33 2024-09-16T10:27:37.826 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:37.826 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T10:27:37.826 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:27:37.826 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:27:37.826 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:38.027 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 32/33 2024-09-16T10:27:38.051 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 32/33 2024-09-16T10:27:38.051 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:38.051 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T10:27:38.051 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:27:38.051 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:27:38.051 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:38.115 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 33/33 2024-09-16T10:27:38.495 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 30/33 2024-09-16T10:27:38.524 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 30/33 2024-09-16T10:27:38.524 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:38.524 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T10:27:38.524 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:27:38.524 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:27:38.524 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 31/33 2024-09-16T10:27:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 31/33 2024-09-16T10:27:39.013 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:39.013 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T10:27:39.013 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:27:39.013 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:27:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 32/33 2024-09-16T10:27:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 32/33 2024-09-16T10:27:39.248 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:39.248 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T10:27:39.249 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:27:39.249 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:27:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:39.309 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 33/33 2024-09-16T10:27:39.996 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 33/33 2024-09-16T10:27:39.996 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 1/33 2024-09-16T10:27:39.996 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 2/33 2024-09-16T10:27:39.996 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 3/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 4/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 5/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-845.ge4e227b3.el9.n 6/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-16T10:27:39.997 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-16T10:27:39.998 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-16T10:27:39.999 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-16T10:27:39.999 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-16T10:27:40.000 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-16T10:27:40.000 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-16T10:27:40.000 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-16T10:27:40.000 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-16T10:27:40.000 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-16T10:27:40.000 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-16T10:27:41.104 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-16T10:27:41.104 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:41.104 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:27:41.104 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:41.104 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:41.104 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:41.104 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:41.104 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:41.104 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T10:27:41.105 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T10:27:41.106 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T10:27:41.107 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T10:27:41.107 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T10:27:41.107 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T10:27:41.107 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:41.107 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:27:41.208 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 33/33 2024-09-16T10:27:41.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 1/33 2024-09-16T10:27:41.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 2/33 2024-09-16T10:27:41.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 3/33 2024-09-16T10:27:41.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 4/33 2024-09-16T10:27:41.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 5/33 2024-09-16T10:27:41.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-845.ge4e227b3.el9.n 6/33 2024-09-16T10:27:41.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-16T10:27:41.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-16T10:27:41.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-16T10:27:41.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-16T10:27:41.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-16T10:27:41.210 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-16T10:27:41.211 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-16T10:27:41.211 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-16T10:27:41.211 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-16T10:27:41.211 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-16T10:27:41.211 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-16T10:27:41.211 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-16T10:27:41.211 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-16T10:27:41.211 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-16T10:27:41.443 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2024-09-16T10:27:42.013 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:49 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:27:42.103 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:27:42.145 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:27:42.146 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T10:27:42.147 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:27:42.219 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2024-09-16T10:27:42.249 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-16T10:27:42.249 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:42.249 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:27:42.249 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:42.249 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T10:27:42.250 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T10:27:42.251 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:42.252 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:27:42.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2024-09-16T10:27:42.786 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:49 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:27:42.918 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 226 k 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:42.919 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 226 k 2024-09-16T10:27:42.920 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 217 k 2024-09-16T10:27:42.920 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:27:43.133 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-18.2.4-845.ge4e227b3.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-09-16T10:27:43.134 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:43.134 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-09-16T10:27:43.134 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:27:43.139 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:27:43.139 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:27:43.142 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:27:43.142 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:27:43.212 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:27:43.244 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:49 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:27:43.334 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:27:43.375 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:27:43.376 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-16T10:27:43.377 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:27:43.443 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2024-09-16T10:27:43.653 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:27:43.735 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:27:44.005 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:50 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:27:44.085 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:27:44.137 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:27:44.138 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:44.138 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:27:44.138 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:44.138 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:27:44.138 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 226 k 2024-09-16T10:27:44.138 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:44.138 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:27:44.139 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:44.139 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-16T10:27:44.139 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:44.139 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 226 k 2024-09-16T10:27:44.139 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 217 k 2024-09-16T10:27:44.139 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:27:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:27:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:44.333 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:27:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:44.333 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:27:44.364 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-18.2.4-845.ge4e227b3.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-09-16T10:27:44.364 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:44.365 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-09-16T10:27:44.365 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:27:44.370 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:27:44.370 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:27:44.373 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:27:44.373 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:27:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:27:44.545 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T10:27:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:27:45.001 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:27:45.119 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:27:45.259 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:27:45.260 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:45.260 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-16T10:27:45.261 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:45.261 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:27:45.261 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 143 k 2024-09-16T10:27:45.262 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:45.262 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:27:45.262 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:45.262 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T10:27:45.263 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:45.263 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 143 k 2024-09-16T10:27:45.263 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 395 k 2024-09-16T10:27:45.263 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:27:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:27:45.454 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-18.2.4-845.ge4e227b 738 kB/s | 143 kB 00:00 2024-09-16T10:27:45.454 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:45.455 INFO:teuthology.orchestra.run.smithi152.stdout:Total 732 kB/s | 143 kB 00:00 2024-09-16T10:27:45.455 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:27:45.462 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:27:45.463 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:27:45.520 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:27:45.521 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:27:45.615 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:27:45.615 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:45.615 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:27:45.615 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:45.615 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:45.615 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:27:45.671 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:27:45.722 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.e 1/1 2024-09-16T10:27:45.747 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.e 1/1 2024-09-16T10:27:45.747 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:45.747 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T10:27:45.747 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:45.852 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T10:27:46.422 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:27:46.554 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:27:46.554 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 143 k 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 143 k 2024-09-16T10:27:46.555 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 395 k 2024-09-16T10:27:46.556 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:27:46.684 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.e 1/1 2024-09-16T10:27:46.684 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:46.684 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:27:46.684 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:46.684 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:46.684 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:27:46.738 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-18.2.4-845.ge4e227b 784 kB/s | 143 kB 00:00 2024-09-16T10:27:46.738 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:46.739 INFO:teuthology.orchestra.run.smithi094.stdout:Total 776 kB/s | 143 kB 00:00 2024-09-16T10:27:46.739 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:27:46.746 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:27:46.746 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:27:46.803 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:27:46.803 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:27:46.915 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2024-09-16T10:27:46.964 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:27:47.072 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.e 1/1 2024-09-16T10:27:47.096 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.e 1/1 2024-09-16T10:27:47.097 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:27:47.097 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T10:27:47.097 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:47.489 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:27:47.585 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:27:47.628 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:27:47.629 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T10:27:47.629 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:27:47.702 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T10:27:48.056 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.e 1/1 2024-09-16T10:27:48.056 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:48.056 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:27:48.056 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:27:48.056 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:48.056 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:27:48.261 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:27:48.301 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2024-09-16T10:27:48.393 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 3.5 M 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 24 k 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 16 k 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T10:27:48.394 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:48.395 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:27:48.395 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:27:48.395 INFO:teuthology.orchestra.run.smithi152.stdout:Install 5 Packages 2024-09-16T10:27:48.395 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:48.395 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.8 M 2024-09-16T10:27:48.395 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 87 M 2024-09-16T10:27:48.395 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:27:48.599 INFO:teuthology.orchestra.run.smithi152.stdout:(1/5): ceph-grafana-dashboards-18.2.4-845.ge4e2 170 kB/s | 24 kB 00:00 2024-09-16T10:27:48.624 INFO:teuthology.orchestra.run.smithi152.stdout:(2/5): ceph-prometheus-alerts-18.2.4-845.ge4e22 97 kB/s | 16 kB 00:00 2024-09-16T10:27:48.666 INFO:teuthology.orchestra.run.smithi152.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 459 kB/s | 31 kB 00:00 2024-09-16T10:27:48.691 INFO:teuthology.orchestra.run.smithi152.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-16T10:27:48.850 INFO:teuthology.orchestra.run.smithi152.stdout:(5/5): ceph-mgr-dashboard-18.2.4-845.ge4e227b3. 8.9 MB/s | 3.5 MB 00:00 2024-09-16T10:27:48.853 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:48.854 INFO:teuthology.orchestra.run.smithi152.stdout:Total 8.3 MB/s | 3.8 MB 00:00 2024-09-16T10:27:48.854 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:27:48.861 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:27:48.861 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:27:48.865 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:27:48.956 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:27:48.956 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:27:48.958 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:27:49.000 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:27:49.002 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-16T10:27:49.002 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:27:49.064 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T10:27:49.209 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:27:49.298 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-16T10:27:49.330 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-16T10:27:49.358 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-845.ge4e227b3.el9.no 3/5 2024-09-16T10:27:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:27:49.760 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 3.5 M 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 24 k 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 16 k 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T10:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:49.762 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:27:49.762 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:27:49.762 INFO:teuthology.orchestra.run.smithi094.stdout:Install 5 Packages 2024-09-16T10:27:49.762 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:49.762 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.8 M 2024-09-16T10:27:49.762 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 87 M 2024-09-16T10:27:49.763 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:27:49.934 INFO:teuthology.orchestra.run.smithi094.stdout:(1/5): ceph-grafana-dashboards-18.2.4-845.ge4e2 173 kB/s | 24 kB 00:00 2024-09-16T10:27:49.959 INFO:teuthology.orchestra.run.smithi094.stdout:(2/5): ceph-prometheus-alerts-18.2.4-845.ge4e22 99 kB/s | 16 kB 00:00 2024-09-16T10:27:50.001 INFO:teuthology.orchestra.run.smithi094.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 460 kB/s | 31 kB 00:00 2024-09-16T10:27:50.026 INFO:teuthology.orchestra.run.smithi094.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-16T10:27:50.210 INFO:teuthology.orchestra.run.smithi094.stdout:(5/5): ceph-mgr-dashboard-18.2.4-845.ge4e227b3. 8.5 MB/s | 3.5 MB 00:00 2024-09-16T10:27:50.214 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:50.214 INFO:teuthology.orchestra.run.smithi094.stdout:Total 8.4 MB/s | 3.8 MB 00:00 2024-09-16T10:27:50.214 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:27:50.221 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:27:50.221 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:27:50.305 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:27:50.306 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:27:50.403 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-845.ge4e227b3.el9.n 4/5 2024-09-16T10:27:50.437 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 5/5 2024-09-16T10:27:50.563 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:27:50.649 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-16T10:27:50.690 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-16T10:27:50.719 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-845.ge4e227b3.el9.no 3/5 2024-09-16T10:27:51.573 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 5/5 2024-09-16T10:27:51.573 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-845.ge4e227b3.el9.n 1/5 2024-09-16T10:27:51.573 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 2/5 2024-09-16T10:27:51.573 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-845.ge4e227b3.el9.no 3/5 2024-09-16T10:27:51.573 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-16T10:27:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-845.ge4e227b3.el9.n 4/5 2024-09-16T10:27:51.797 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 5/5 2024-09-16T10:27:51.827 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-16T10:27:51.827 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:51.827 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:27:51.827 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:51.827 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:51.827 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:51.827 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T10:27:51.828 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T10:27:51.828 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:51.828 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:27:52.059 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T10:27:52.621 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:27:52.753 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:27:52.754 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-09-16T10:27:52.754 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 7.4 M 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T10:27:52.755 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T10:27:52.756 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T10:27:52.756 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T10:27:52.756 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T10:27:52.756 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:52.756 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:27:52.756 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-09-16T10:27:52.756 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-09-16T10:27:52.756 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:27:52.757 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 38 M 2024-09-16T10:27:52.757 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 212 M 2024-09-16T10:27:52.757 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:27:52.904 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 5/5 2024-09-16T10:27:52.904 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-845.ge4e227b3.el9.n 1/5 2024-09-16T10:27:52.904 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 2/5 2024-09-16T10:27:52.904 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-845.ge4e227b3.el9.no 3/5 2024-09-16T10:27:52.904 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-16T10:27:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-16T10:27:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:53.162 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:27:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:27:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T10:27:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T10:27:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:53.163 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:27:53.165 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-16T10:27:53.249 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.3 MB/s | 801 kB 00:00 2024-09-16T10:27:53.274 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-16T10:27:53.316 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-16T10:27:53.366 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-16T10:27:53.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T10:27:53.450 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.4-84 17 MB/s | 7.4 MB 00:00 2024-09-16T10:27:53.575 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.2 MB/s | 3.0 MB 00:00 2024-09-16T10:27:53.650 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2024-09-16T10:27:53.676 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-16T10:27:53.701 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.9 MB/s | 250 kB 00:00 2024-09-16T10:27:53.784 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 5.1 MB 00:00 2024-09-16T10:27:53.947 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:27:54.018 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-16T10:27:54.019 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:54.019 INFO:teuthology.orchestra.run.smithi152.stdout:Total 30 MB/s | 38 MB 00:01 2024-09-16T10:27:54.078 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:27:54.079 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-09-16T10:27:54.080 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:27:54.080 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-09-16T10:27:54.080 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:27:54.080 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 7.4 M 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T10:27:54.081 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T10:27:54.082 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T10:27:54.082 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:54.082 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:27:54.082 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-09-16T10:27:54.082 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-09-16T10:27:54.082 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:27:54.082 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 38 M 2024-09-16T10:27:54.083 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 212 M 2024-09-16T10:27:54.083 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:27:54.368 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:27:54.394 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:27:54.394 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:27:54.460 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-16T10:27:54.502 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-16T10:27:54.560 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-09-16T10:27:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-09-16T10:27:54.661 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 659 kB/s | 38 kB 00:00 2024-09-16T10:27:54.795 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.4-84 15 MB/s | 7.4 MB 00:00 2024-09-16T10:27:54.893 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:27:54.893 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:27:54.937 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-09-16T10:27:55.270 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.9 MB/s | 3.0 MB 00:00 2024-09-16T10:27:55.371 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 250 kB 00:00 2024-09-16T10:27:55.653 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:27:55.724 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T10:27:55.766 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:27:55.867 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:27:55.900 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T10:27:55.955 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.8 MB/s | 4.9 MB 00:01 2024-09-16T10:27:56.084 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T10:27:56.116 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T10:27:56.154 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-09-16T10:27:56.201 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T10:27:56.876 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T10:27:57.004 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T10:27:58.274 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:27:59.306 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T10:27:59.339 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227 12/12 2024-09-16T10:27:59.629 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.8 MB/s | 16 MB 00:04 2024-09-16T10:27:59.630 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:27:59.631 INFO:teuthology.orchestra.run.smithi094.stdout:Total 6.9 MB/s | 38 MB 00:05 2024-09-16T10:27:59.990 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:00.016 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:00.017 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:00.519 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:00.519 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:01.280 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:01.354 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T10:28:01.394 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:28:01.501 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:28:01.537 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T10:28:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T10:28:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T10:28:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T10:28:01.953 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227 12/12 2024-09-16T10:28:01.953 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227 1/12 2024-09-16T10:28:01.953 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:28:01.953 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:28:01.953 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T10:28:01.953 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T10:28:01.953 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T10:28:01.953 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T10:28:01.954 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T10:28:01.954 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T10:28:01.954 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:28:01.954 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T10:28:02.348 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T10:28:02.348 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:02.348 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:02.348 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:28:02.348 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T10:28:02.348 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:02.349 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:02.495 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T10:28:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T10:28:02.746 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2024-09-16T10:28:03.320 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:10 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:03.452 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:03.453 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:03.453 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:28:03.453 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 49 k 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T10:28:03.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T10:28:03.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T10:28:03.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T10:28:03.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T10:28:03.455 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:03.455 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:28:03.455 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:03.455 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-09-16T10:28:03.455 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:03.456 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.2 M 2024-09-16T10:28:03.456 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-09-16T10:28:03.456 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:28:03.852 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:28:03.871 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 26 kB 00:00 2024-09-16T10:28:03.897 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 19 kB 00:00 2024-09-16T10:28:03.922 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): ceph-mgr-rook-18.2.4-845.ge4e227b3.el9. 278 kB/s | 49 kB 00:00 2024-09-16T10:28:03.989 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-16T10:28:04.039 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 1.1 MB/s | 54 kB 00:00 2024-09-16T10:28:04.081 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 862 kB/s | 159 kB 00:00 2024-09-16T10:28:04.131 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 350 kB/s | 32 kB 00:00 2024-09-16T10:28:04.157 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 191 kB/s | 14 kB 00:00 2024-09-16T10:28:04.182 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.0 MB/s | 279 kB 00:00 2024-09-16T10:28:04.207 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-16T10:28:04.249 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-16T10:28:04.283 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-09-16T10:28:04.299 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-09-16T10:28:04.302 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:04.302 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-09-16T10:28:04.365 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:28:04.380 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:28:04.380 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:28:04.467 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:28:04.469 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:28:04.874 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T10:28:04.912 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227 12/12 2024-09-16T10:28:04.965 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:28:05.132 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T10:28:05.184 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T10:28:05.240 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T10:28:05.277 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T10:28:05.318 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T10:28:05.420 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T10:28:05.530 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T10:28:05.586 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T10:28:06.304 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T10:28:06.358 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T10:28:06.392 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T10:28:06.440 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T10:28:06.470 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 13/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 13/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 1/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T10:28:07.306 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T10:28:07.307 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T10:28:07.307 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T10:28:07.555 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227 12/12 2024-09-16T10:28:07.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227 1/12 2024-09-16T10:28:07.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:28:07.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:28:07.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T10:28:07.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T10:28:07.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T10:28:07.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T10:28:07.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T10:28:07.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T10:28:07.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:28:07.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T10:28:07.634 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T10:28:07.634 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:07.634 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:07.634 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:28:07.634 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T10:28:07.634 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T10:28:07.634 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T10:28:07.634 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:07.635 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:07.887 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T10:28:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T10:28:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:07.995 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:28:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T10:28:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T10:28:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T10:28:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T10:28:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T10:28:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T10:28:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T10:28:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T10:28:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T10:28:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T10:28:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T10:28:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:07.996 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:08.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2024-09-16T10:28:08.462 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:15 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:08.595 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:08.595 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:08.595 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:28:08.596 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:08.596 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:28:08.596 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 139 k 2024-09-16T10:28:08.596 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:28:08.596 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T10:28:08.596 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T10:28:08.596 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T10:28:08.596 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T10:28:08.597 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T10:28:08.597 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:08.597 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:28:08.597 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:08.597 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-09-16T10:28:08.597 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:08.598 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 7.0 M 2024-09-16T10:28:08.599 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 33 M 2024-09-16T10:28:08.599 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:28:08.966 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:14 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:08.984 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): ceph-mgr-cephadm-18.2.4-845.ge4e227b3.el 582 kB/s | 139 kB 00:00 2024-09-16T10:28:09.034 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-16T10:28:09.060 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-16T10:28:09.085 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T10:28:09.106 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:09.107 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:09.107 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 49 k 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T10:28:09.108 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T10:28:09.109 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T10:28:09.109 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T10:28:09.109 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T10:28:09.109 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T10:28:09.109 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:09.109 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:28:09.109 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:09.109 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-09-16T10:28:09.109 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:09.110 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.2 M 2024-09-16T10:28:09.110 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-09-16T10:28:09.110 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:28:09.243 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 498 kB/s | 248 kB 00:00 2024-09-16T10:28:09.422 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 209 kB/s | 19 kB 00:00 2024-09-16T10:28:09.447 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 219 kB/s | 26 kB 00:00 2024-09-16T10:28:09.517 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-09-16T10:28:09.547 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-09-16T10:28:09.573 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): ceph-mgr-rook-18.2.4-845.ge4e227b3.el9. 203 kB/s | 49 kB 00:00 2024-09-16T10:28:09.598 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T10:28:09.657 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 385 kB/s | 32 kB 00:00 2024-09-16T10:28:09.682 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-09-16T10:28:09.707 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 129 kB/s | 14 kB 00:00 2024-09-16T10:28:09.741 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-16T10:28:09.766 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-16T10:28:09.791 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T10:28:09.825 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-16T10:28:09.828 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:09.828 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-16T10:28:09.894 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:09.909 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:09.909 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:10.001 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:10.001 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:10.344 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.7 MB/s | 6.0 MB 00:01 2024-09-16T10:28:10.345 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:10.346 INFO:teuthology.orchestra.run.smithi152.stdout:Total 4.0 MB/s | 7.0 MB 00:01 2024-09-16T10:28:10.418 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:28:10.429 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:28:10.429 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:28:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:10.528 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:28:10.528 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:28:10.672 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T10:28:10.726 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T10:28:10.790 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T10:28:10.828 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T10:28:10.844 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:28:10.869 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T10:28:10.970 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T10:28:11.019 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T10:28:11.062 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T10:28:11.081 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T10:28:11.129 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T10:28:11.580 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T10:28:11.664 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T10:28:11.734 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T10:28:11.757 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 6/6 2024-09-16T10:28:11.840 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T10:28:11.894 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T10:28:11.927 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T10:28:11.975 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T10:28:11.997 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 13/13 2024-09-16T10:28:12.586 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 6/6 2024-09-16T10:28:12.586 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/6 2024-09-16T10:28:12.586 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T10:28:12.587 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T10:28:12.587 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T10:28:12.587 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T10:28:12.854 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 13/13 2024-09-16T10:28:12.854 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 1/13 2024-09-16T10:28:12.854 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T10:28:12.854 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T10:28:12.854 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T10:28:12.854 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T10:28:12.854 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T10:28:12.854 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T10:28:12.855 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T10:28:12.855 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T10:28:12.855 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T10:28:12.855 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T10:28:12.855 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T10:28:12.855 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T10:28:12.855 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:12.856 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:12.856 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:28:12.856 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T10:28:12.856 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T10:28:12.856 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T10:28:12.856 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T10:28:12.856 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T10:28:12.856 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:12.856 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:13.107 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2024-09-16T10:28:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T10:28:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:13.177 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T10:28:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T10:28:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:13.179 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:13.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T10:28:13.673 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:20 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:13.804 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:13.804 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:13.804 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 852 k 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-09-16T10:28:13.805 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:13.806 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 932 k 2024-09-16T10:28:13.806 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 2.7 M 2024-09-16T10:28:13.806 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:28:14.045 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 776 kB/s | 80 kB 00:00 2024-09-16T10:28:14.053 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:20 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:14.186 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 139 k 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T10:28:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T10:28:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T10:28:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:14.188 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:28:14.188 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:14.188 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-09-16T10:28:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:14.188 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 7.0 M 2024-09-16T10:28:14.189 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 33 M 2024-09-16T10:28:14.189 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:28:14.237 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-fuse-18.2.4-845.ge4e227b3.el9.x86_6 2.8 MB/s | 852 kB 00:00 2024-09-16T10:28:14.237 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:14.237 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.1 MB/s | 932 kB 00:00 2024-09-16T10:28:14.249 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:28:14.270 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:28:14.270 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:28:14.349 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:28:14.349 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:28:14.529 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): ceph-mgr-cephadm-18.2.4-845.ge4e227b3.el 710 kB/s | 139 kB 00:00 2024-09-16T10:28:14.537 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:28:14.554 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-16T10:28:14.613 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-16T10:28:14.638 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-16T10:28:14.664 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 529 kB/s | 58 kB 00:00 2024-09-16T10:28:14.676 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T10:28:14.771 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 2/2 2024-09-16T10:28:15.451 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 2/2 2024-09-16T10:28:15.452 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/2 2024-09-16T10:28:15.729 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T10:28:15.729 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:15.729 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:15.729 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T10:28:15.729 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:15.729 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:16.000 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-volume 2024-09-16T10:28:16.577 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:23 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:16.689 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.5 MB/s | 6.0 MB 00:02 2024-09-16T10:28:16.691 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:16.692 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.8 MB/s | 7.0 MB 00:02 2024-09-16T10:28:16.715 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:16.716 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:16.716 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T10:28:16.716 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:16.716 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:28:16.716 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 264 k 2024-09-16T10:28:16.716 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T10:28:16.717 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T10:28:16.717 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:16.717 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:28:16.717 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:16.717 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-09-16T10:28:16.717 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:16.717 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 342 k 2024-09-16T10:28:16.718 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 1.5 M 2024-09-16T10:28:16.718 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:28:16.766 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:16.777 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:16.778 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:16.832 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 826 kB/s | 77 kB 00:00 2024-09-16T10:28:16.877 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:16.880 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:16.982 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-volume-18.2.4-845.ge4e227b3.el9.noa 1.1 MB/s | 264 kB 00:00 2024-09-16T10:28:16.983 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:16.983 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.3 MB/s | 342 kB 00:00 2024-09-16T10:28:16.995 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:28:17.004 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:28:17.004 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:28:17.040 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:28:17.040 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:28:17.190 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:28:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:17.375 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-16T10:28:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T10:28:17.418 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 2/2 2024-09-16T10:28:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T10:28:17.434 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 2/2 2024-09-16T10:28:17.434 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:28:17.434 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T10:28:17.435 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:17.956 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T10:28:18.035 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T10:28:18.097 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T10:28:18.120 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 6/6 2024-09-16T10:28:18.177 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 1/2 2024-09-16T10:28:18.501 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-16T10:28:18.501 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:18.502 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:18.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:28:18.502 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T10:28:18.502 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:18.502 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:18.804 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2024-09-16T10:28:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 6/6 2024-09-16T10:28:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/6 2024-09-16T10:28:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T10:28:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T10:28:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T10:28:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T10:28:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T10:28:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:19.187 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:28:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T10:28:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T10:28:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T10:28:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T10:28:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T10:28:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:19.189 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:19.373 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:19.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2024-09-16T10:28:19.509 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:19.509 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 127 k 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 127 k 2024-09-16T10:28:19.510 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 456 k 2024-09-16T10:28:19.511 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:28:19.721 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-18.2.4-845.ge4e227b3.el9.x86_64. 602 kB/s | 127 kB 00:00 2024-09-16T10:28:19.722 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:19.722 INFO:teuthology.orchestra.run.smithi152.stdout:Total 598 kB/s | 127 kB 00:00 2024-09-16T10:28:19.722 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:28:19.727 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:28:19.727 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:28:19.790 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:28:19.790 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:28:19.945 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:28:19.996 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:20.043 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:20.132 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:20.132 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 852 k 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 932 k 2024-09-16T10:28:20.133 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.7 M 2024-09-16T10:28:20.134 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:28:20.300 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 554 kB/s | 80 kB 00:00 2024-09-16T10:28:20.442 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-18.2.4-845.ge4e227b3.el9.x86_6 2.9 MB/s | 852 kB 00:00 2024-09-16T10:28:20.443 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:20.443 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.9 MB/s | 932 kB 00:00 2024-09-16T10:28:20.454 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:20.476 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:20.476 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:20.476 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:20.557 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:20.558 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:20.737 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:20.737 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:20.737 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:20.737 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:20.737 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:20.737 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T10:28:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 2/2 2024-09-16T10:28:21.004 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2024-09-16T10:28:21.580 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:21.663 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 2/2 2024-09-16T10:28:21.664 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/2 2024-09-16T10:28:21.671 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:21.715 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:21.716 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T10:28:21.717 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:21.820 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2024-09-16T10:28:21.914 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T10:28:21.914 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:21.914 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:21.915 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T10:28:21.915 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:21.915 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:22.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-volume 2024-09-16T10:28:22.381 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:22.514 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:22.514 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:22.514 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 31 k 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 31 k 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 139 k 2024-09-16T10:28:22.515 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:28:22.673 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-18.2.4-845.ge4e227b3.el9.x86_64 196 kB/s | 31 kB 00:00 2024-09-16T10:28:22.674 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:22.674 INFO:teuthology.orchestra.run.smithi152.stdout:Total 195 kB/s | 31 kB 00:00 2024-09-16T10:28:22.674 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:28:22.678 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:28:22.678 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:28:22.695 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:28:22.695 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:28:22.776 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:28:22.821 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:22.869 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:22.961 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:22.962 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-16T10:28:22.963 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:22.963 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:28:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 264 k 2024-09-16T10:28:22.963 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-16T10:28:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T10:28:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:22.964 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:28:22.964 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:22.964 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-09-16T10:28:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:22.965 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 342 k 2024-09-16T10:28:22.965 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.5 M 2024-09-16T10:28:22.965 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:28:23.252 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:23.307 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 863 kB/s | 77 kB 00:00 2024-09-16T10:28:23.465 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-volume-18.2.4-845.ge4e227b3.el9.noa 1.0 MB/s | 264 kB 00:00 2024-09-16T10:28:23.466 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:23.466 INFO:teuthology.orchestra.run.smithi094.stdout:Total 678 kB/s | 342 kB 00:00 2024-09-16T10:28:23.478 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:23.488 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:23.488 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:23.492 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:23.493 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:23.493 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:23.493 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:23.493 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:23.493 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:23.528 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:23.528 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:23.670 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2024-09-16T10:28:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:23.925 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-16T10:28:23.960 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 2/2 2024-09-16T10:28:23.975 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 2/2 2024-09-16T10:28:23.975 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:28:23.976 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T10:28:23.976 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:24.237 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:24.327 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:24.370 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:24.371 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T10:28:24.371 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:24.442 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2024-09-16T10:28:24.681 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 1/2 2024-09-16T10:28:24.995 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-16T10:28:24.995 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:24.995 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:24.995 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:28:24.995 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T10:28:24.995 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:24.995 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:25.010 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:25.101 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:25.143 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:25.144 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T10:28:25.145 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:25.217 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2024-09-16T10:28:25.316 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2024-09-16T10:28:25.783 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:25.873 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:25.896 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:25.917 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:25.918 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T10:28:25.918 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:25.991 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2024-09-16T10:28:26.033 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:26.034 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-16T10:28:26.034 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:26.034 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:28:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 127 k 2024-09-16T10:28:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:26.034 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:28:26.034 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:26.035 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-16T10:28:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:26.035 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 127 k 2024-09-16T10:28:26.035 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 456 k 2024-09-16T10:28:26.035 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:28:26.223 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-18.2.4-845.ge4e227b3.el9.x86_64. 675 kB/s | 127 kB 00:00 2024-09-16T10:28:26.224 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:26.224 INFO:teuthology.orchestra.run.smithi094.stdout:Total 670 kB/s | 127 kB 00:00 2024-09-16T10:28:26.224 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:26.229 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:26.230 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:26.292 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:26.293 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:26.553 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:26.554 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:26.643 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:26.686 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:26.687 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T10:28:26.687 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:26.757 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2024-09-16T10:28:27.021 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:27.299 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:27.299 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:27.299 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:27.299 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:27.300 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:27.300 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:27.324 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:27.413 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:27.455 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:27.456 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T10:28:27.456 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:27.524 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2024-09-16T10:28:27.526 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2024-09-16T10:28:28.095 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:28.105 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:28.184 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:28.197 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:28.226 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:28.227 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T10:28:28.227 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:28.240 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:28.241 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-16T10:28:28.242 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:28.299 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2024-09-16T10:28:28.326 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2024-09-16T10:28:28.871 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:29.005 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 87 k 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 87 k 2024-09-16T10:28:29.006 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 227 k 2024-09-16T10:28:29.007 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:28:29.027 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:29.027 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:29.027 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-16T10:28:29.027 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:29.027 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:28:29.027 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 31 k 2024-09-16T10:28:29.028 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:29.028 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:28:29.028 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:29.028 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-16T10:28:29.028 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:29.028 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 31 k 2024-09-16T10:28:29.029 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 139 k 2024-09-16T10:28:29.029 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:28:29.177 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-18.2.4-845.ge4e227b3.el9.x86_64 208 kB/s | 31 kB 00:00 2024-09-16T10:28:29.178 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:29.178 INFO:teuthology.orchestra.run.smithi094.stdout:Total 207 kB/s | 31 kB 00:00 2024-09-16T10:28:29.178 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:29.182 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:29.182 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:29.201 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-18.2.4-845.ge4e227b3.el9.x86_64.rpm 446 kB/s | 87 kB 00:00 2024-09-16T10:28:29.202 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:29.202 INFO:teuthology.orchestra.run.smithi152.stdout:Total 443 kB/s | 87 kB 00:00 2024-09-16T10:28:29.202 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:28:29.205 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:29.205 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:29.209 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:28:29.209 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:28:29.277 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:28:29.277 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:28:29.291 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:29.398 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:29.432 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:28:29.523 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:29.765 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:29.936 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:29.991 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:29.991 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:30.202 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:30.202 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:30.202 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:30.202 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:30.202 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:30.202 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:30.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2024-09-16T10:28:30.484 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-09-16T10:28:30.812 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:30.901 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:30.944 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:30.945 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-16T10:28:30.945 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:31.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2024-09-16T10:28:31.059 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:31.197 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 3.0 M 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T10:28:31.198 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:31.199 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.0 M 2024-09-16T10:28:31.199 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 12 M 2024-09-16T10:28:31.199 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:28:31.546 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-18.2.4-845.ge4e227b3.el9.x86_64.rpm 8.6 MB/s | 3.0 MB 00:00 2024-09-16T10:28:31.546 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:31.547 INFO:teuthology.orchestra.run.smithi152.stdout:Total 8.6 MB/s | 3.0 MB 00:00 2024-09-16T10:28:31.547 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:28:31.556 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:28:31.556 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:28:31.586 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:31.637 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:28:31.637 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:28:31.677 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:31.720 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:31.721 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-16T10:28:31.721 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:31.793 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2024-09-16T10:28:32.067 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:28:32.130 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:32.155 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:32.155 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:28:32.155 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T10:28:32.155 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:28:32.156 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:28:32.156 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:32.352 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:32.442 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:32.484 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:32.485 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-16T10:28:32.485 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:32.570 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2024-09-16T10:28:33.133 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:33.222 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:33.264 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:33.265 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-16T10:28:33.265 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:33.281 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:33.281 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:33.281 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:33.281 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:33.281 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:33.282 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:33.344 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2024-09-16T10:28:33.565 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2024-09-16T10:28:33.913 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:34.003 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:34.046 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:34.047 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-16T10:28:34.048 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:34.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2024-09-16T10:28:34.145 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 10:26:53 AM UTC. 2024-09-16T10:28:34.282 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T10:28:34.283 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:34.283 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-09-16T10:28:34.283 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:34.283 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T10:28:34.284 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 172 k 2024-09-16T10:28:34.284 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:34.284 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T10:28:34.284 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T10:28:34.284 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T10:28:34.284 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:34.284 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 172 k 2024-09-16T10:28:34.284 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 494 k 2024-09-16T10:28:34.284 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T10:28:34.689 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:34.757 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-18.2.4-845.ge4e227b3.el9.x86_64.rpm 364 kB/s | 172 kB 00:00 2024-09-16T10:28:34.758 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:34.758 INFO:teuthology.orchestra.run.smithi152.stdout:Total 362 kB/s | 172 kB 00:00 2024-09-16T10:28:34.758 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T10:28:34.765 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T10:28:34.765 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T10:28:34.778 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:28:34.821 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:34.822 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-16T10:28:34.822 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:34.835 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T10:28:34.835 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T10:28:34.895 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2024-09-16T10:28:34.998 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T10:28:35.104 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:35.454 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:35.521 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:35.586 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:35.586 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-16T10:28:35.586 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:35.586 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:28:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 87 k 2024-09-16T10:28:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:35.587 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:28:35.587 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:35.587 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-16T10:28:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:35.587 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 87 k 2024-09-16T10:28:35.588 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 227 k 2024-09-16T10:28:35.588 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:28:35.790 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-18.2.4-845.ge4e227b3.el9.x86_64.rpm 430 kB/s | 87 kB 00:00 2024-09-16T10:28:35.790 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:35.790 INFO:teuthology.orchestra.run.smithi094.stdout:Total 427 kB/s | 87 kB 00:00 2024-09-16T10:28:35.791 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:35.797 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:35.797 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:35.808 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:35.808 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:35.808 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T10:28:35.809 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:35.809 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:35.809 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T10:28:35.864 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:35.864 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:36.053 DEBUG:teuthology.parallel:result is None 2024-09-16T10:28:36.120 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:36.519 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:36.770 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:36.770 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:36.770 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:36.770 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:36.770 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:36.770 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:37.042 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2024-09-16T10:28:37.615 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:37.748 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:37.748 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:37.748 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-16T10:28:37.748 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:37.748 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:28:37.748 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 3.0 M 2024-09-16T10:28:37.749 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:37.749 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:28:37.749 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:37.749 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-16T10:28:37.749 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:37.749 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.0 M 2024-09-16T10:28:37.750 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 12 M 2024-09-16T10:28:37.750 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:28:38.587 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-18.2.4-845.ge4e227b3.el9.x86_64.rpm 3.6 MB/s | 3.0 MB 00:00 2024-09-16T10:28:38.588 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:38.588 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.6 MB/s | 3.0 MB 00:00 2024-09-16T10:28:38.588 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:38.597 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:38.597 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:38.678 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:38.678 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:39.280 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:39.306 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:39.306 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:28:39.306 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T10:28:39.306 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:28:39.306 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:28:39.307 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:40.385 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:40.385 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:40.385 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:40.385 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:40.385 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:40.385 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:40.635 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-09-16T10:28:41.222 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 10:26:54 AM UTC. 2024-09-16T10:28:41.357 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-16T10:28:41.358 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-16T10:28:41.358 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:41.358 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-16T10:28:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 172 k 2024-09-16T10:28:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:41.359 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-16T10:28:41.359 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-16T10:28:41.359 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-16T10:28:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:41.359 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 172 k 2024-09-16T10:28:41.359 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 494 k 2024-09-16T10:28:41.359 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-16T10:28:41.726 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-18.2.4-845.ge4e227b3.el9.x86_64.rpm 469 kB/s | 172 kB 00:00 2024-09-16T10:28:41.726 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:28:41.726 INFO:teuthology.orchestra.run.smithi094.stdout:Total 467 kB/s | 172 kB 00:00 2024-09-16T10:28:41.727 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-16T10:28:41.734 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-16T10:28:41.734 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-16T10:28:41.804 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-16T10:28:41.804 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-16T10:28:41.970 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-16T10:28:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:42.826 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:28:42.826 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:42.826 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-16T10:28:42.826 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:28:42.826 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:42.826 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-16T10:28:43.085 DEBUG:teuthology.parallel:result is None 2024-09-16T10:28:43.085 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:28:43.229 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T10:28:43.251 INFO:teuthology.orchestra.run.smithi094.stdout:18.2.4-845.ge4e227b3.el9 2024-09-16T10:28:43.252 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845.ge4e227b3.el9 2024-09-16T10:28:43.252 INFO:teuthology.task.install:The correct ceph version 18.2.4-845.ge4e227b3 is installed. 2024-09-16T10:28:43.254 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:28:43.396 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T10:28:43.422 INFO:teuthology.orchestra.run.smithi152.stdout:18.2.4-845.ge4e227b3.el9 2024-09-16T10:28:43.423 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845.ge4e227b3.el9 2024-09-16T10:28:43.423 INFO:teuthology.task.install:The correct ceph version 18.2.4-845.ge4e227b3 is installed. 2024-09-16T10:28:43.425 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T10:28:43.425 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:43.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T10:28:43.450 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:28:43.451 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T10:28:43.494 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T10:28:43.494 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:43.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T10:28:43.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T10:28:43.582 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:28:43.583 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T10:28:43.611 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T10:28:43.679 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T10:28:43.680 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:43.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T10:28:43.704 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T10:28:43.769 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:28:43.769 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T10:28:43.796 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T10:28:43.862 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T10:28:43.863 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:43.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T10:28:43.913 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T10:28:43.977 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:28:43.977 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T10:28:44.003 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T10:28:44.069 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T10:28:44.149 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T10:28:44.149 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2024-09-16T10:28:44.151 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /var/log/ceph 2024-09-16T10:28:44.177 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T10:28:44.177 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T10:28:44.216 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T10:28:44.242 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T10:28:44.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T10:28:44.280 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T10:28:44.317 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T10:28:44.317 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-16T10:28:44.318 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655', 'branch': 'reef', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906655', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'mon_bind_msgr2': False, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 931, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1390, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'sleep_before_teardown': 0, 'subset': '5319/100000', 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6', 'targets': {'smithi094.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMEbhcnxotVjRvNXM4TQfLe90v6fM8oMlstd1MndhTfk6Ee+NUOdeja0vhjcKFi+2jKG7g1E5Z98W6dJ1yZWkGI=', 'smithi152.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjIz7uKVJ6cOLwfgH8Ei+hjNgGTTYcaUn9tmkjMRVNJhN9SQ7OeOO2P0fHtDQwYDkqu9AXipMF9I39ZI/frz74='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_22:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T10:28:44.318 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T10:28:44.345 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T10:28:44.363 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T10:28:44.399 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T10:28:44.427 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:44.427 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T10:28:44.478 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T10:28:44.479 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-09-16T10:28:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T10:28:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:28:44.534 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-16T10:28:44.534 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:28:44.534 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:28:44.534 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-16 10:28:39.559533352 +0000 2024-09-16T10:28:44.535 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-16 10:26:11.917325120 +0000 2024-09-16T10:28:44.535 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-16 10:26:11.917325120 +0000 2024-09-16T10:28:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-16 10:26:11.917325120 +0000 2024-09-16T10:28:44.535 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T10:28:44.596 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-16T10:28:44.596 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-16T10:28:44.596 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 8.3808e-05 s, 6.1 MB/s 2024-09-16T10:28:44.597 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T10:28:44.653 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-09-16T10:28:44.708 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T10:28:44.709 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:28:44.709 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T10:28:44.709 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:28:44.709 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:28:44.709 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-16 10:28:39.559533352 +0000 2024-09-16T10:28:44.709 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-16 10:26:12.235320082 +0000 2024-09-16T10:28:44.709 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-16 10:26:12.235320082 +0000 2024-09-16T10:28:44.709 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-16 10:26:12.235320082 +0000 2024-09-16T10:28:44.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T10:28:44.772 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-16T10:28:44.772 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-16T10:28:44.772 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000108179 s, 4.7 MB/s 2024-09-16T10:28:44.773 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T10:28:44.829 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-09-16T10:28:44.884 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T10:28:44.885 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:28:44.885 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T10:28:44.885 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:28:44.885 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:28:44.885 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-16 10:28:39.559533352 +0000 2024-09-16T10:28:44.885 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-16 10:26:12.545315172 +0000 2024-09-16T10:28:44.885 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-16 10:26:12.545315172 +0000 2024-09-16T10:28:44.885 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-16 10:26:12.545315172 +0000 2024-09-16T10:28:44.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T10:28:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-16T10:28:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-16T10:28:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 9.8007e-05 s, 5.2 MB/s 2024-09-16T10:28:44.948 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T10:28:45.004 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-09-16T10:28:45.059 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T10:28:45.060 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:28:45.060 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T10:28:45.060 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:28:45.060 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:28:45.060 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-16 10:28:39.560533336 +0000 2024-09-16T10:28:45.060 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-16 10:26:12.865310103 +0000 2024-09-16T10:28:45.060 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-16 10:26:12.865310103 +0000 2024-09-16T10:28:45.060 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-16 10:26:12.865310103 +0000 2024-09-16T10:28:45.060 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T10:28:45.122 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-16T10:28:45.122 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-16T10:28:45.122 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 8.2143e-05 s, 6.2 MB/s 2024-09-16T10:28:45.123 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T10:28:45.180 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:28:45.180 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:28:45.180 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T10:28:45.199 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T10:28:45.199 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-09-16T10:28:45.257 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T10:28:45.257 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:28:45.257 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T10:28:45.257 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:28:45.258 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:28:45.258 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-16 10:28:32.430159889 +0000 2024-09-16T10:28:45.258 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-16 10:26:11.981319331 +0000 2024-09-16T10:28:45.258 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-16 10:26:11.981319331 +0000 2024-09-16T10:28:45.258 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-16 10:26:11.981319331 +0000 2024-09-16T10:28:45.258 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T10:28:45.324 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-16T10:28:45.324 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-16T10:28:45.324 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000212465 s, 2.4 MB/s 2024-09-16T10:28:45.325 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T10:28:45.384 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-09-16T10:28:45.441 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T10:28:45.441 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:28:45.441 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-16T10:28:45.441 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:28:45.441 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:28:45.441 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-16 10:28:32.430159889 +0000 2024-09-16T10:28:45.442 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-16 10:26:12.325315591 +0000 2024-09-16T10:28:45.442 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-16 10:26:12.325315591 +0000 2024-09-16T10:28:45.442 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-16 10:26:12.325315591 +0000 2024-09-16T10:28:45.442 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T10:28:45.507 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-16T10:28:45.507 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-16T10:28:45.507 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000219656 s, 2.3 MB/s 2024-09-16T10:28:45.508 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T10:28:45.566 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-09-16T10:28:45.624 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T10:28:45.624 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:28:45.624 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T10:28:45.624 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:28:45.624 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:28:45.624 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-16 10:28:32.430159889 +0000 2024-09-16T10:28:45.624 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-16 10:26:12.648312079 +0000 2024-09-16T10:28:45.624 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-16 10:26:12.648312079 +0000 2024-09-16T10:28:45.624 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-16 10:26:12.648312079 +0000 2024-09-16T10:28:45.625 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T10:28:45.690 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-16T10:28:45.690 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-16T10:28:45.690 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000240659 s, 2.1 MB/s 2024-09-16T10:28:45.691 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T10:28:45.748 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-09-16T10:28:45.804 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T10:28:45.805 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:28:45.805 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T10:28:45.805 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:28:45.805 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:28:45.805 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-16 10:28:32.431159876 +0000 2024-09-16T10:28:45.805 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-16 10:26:12.976308513 +0000 2024-09-16T10:28:45.805 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-16 10:26:12.976308513 +0000 2024-09-16T10:28:45.805 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-16 10:26:12.976308513 +0000 2024-09-16T10:28:45.805 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T10:28:45.869 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-16T10:28:45.869 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-16T10:28:45.869 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000233219 s, 2.2 MB/s 2024-09-16T10:28:45.870 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T10:28:45.927 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T10:28:45.927 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T10:28:45.927 INFO:tasks.ceph:Generating config... 2024-09-16T10:28:45.928 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T10:28:45.928 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T10:28:45.928 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T10:28:45.929 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T10:28:45.929 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-16T10:28:45.929 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-16T10:28:45.929 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-16T10:28:45.929 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-16T10:28:45.929 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-16T10:28:45.929 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-16T10:28:45.929 INFO:tasks.ceph:[global] ms type = async 2024-09-16T10:28:45.929 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T10:28:45.929 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-16T10:28:45.929 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T10:28:45.929 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T10:28:45.929 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T10:28:45.929 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] paxos min = 10 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T10:28:45.930 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-09-16T10:28:45.930 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T10:28:45.930 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T10:28:45.930 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T10:28:45.931 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T10:28:45.932 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T10:28:45.932 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T10:28:45.932 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T10:28:45.932 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T10:28:45.932 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T10:28:45.932 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T10:28:45.932 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T10:28:45.987 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T10:28:45.990 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T10:28:46.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T10:28:46.098 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.94:6789]'), ('mon.b', '[v1:172.21.15.94:6790]'), ('mon.c', '[v1:172.21.15.94:6791]'), ('mon.d', '[v1:172.21.15.94:6792]'), ('mon.e', '[v1:172.21.15.94:6793]'), ('mon.f', '[v1:172.21.15.152:6789]'), ('mon.g', '[v1:172.21.15.152:6790]'), ('mon.h', '[v1:172.21.15.152:6791]'), ('mon.i', '[v1:172.21.15.152:6792]')] 2024-09-16T10:28:46.099 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.94:6789],[v1:172.21.15.94:6790],[v1:172.21.15.94:6791],[v1:172.21.15.94:6792],[v1:172.21.15.94:6793],[v1:172.21.15.152:6789],[v1:172.21.15.152:6790],[v1:172.21.15.152:6791],[v1:172.21.15.152:6792]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2024-09-16T10:28:46.100 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:46.100 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T10:28:46.155 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.94:6789]' --addv b '[v1:172.21.15.94:6790]' --addv c '[v1:172.21.15.94:6791]' --addv d '[v1:172.21.15.94:6792]' --addv e '[v1:172.21.15.94:6793]' --addv f '[v1:172.21.15.152:6789]' --addv g '[v1:172.21.15.152:6790]' --addv h '[v1:172.21.15.152:6791]' --addv i '[v1:172.21.15.152:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:28:46.258 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:28:46.258 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: generated fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:28:46.258 INFO:teuthology.orchestra.run.smithi094.stdout:setting min_mon_release = pacific 2024-09-16T10:28:46.258 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 0 2024-09-16T10:28:46.258 INFO:teuthology.orchestra.run.smithi094.stdout:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:28:46.258 INFO:teuthology.orchestra.run.smithi094.stdout:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:min_mon_release 16 (pacific) 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:election_strategy: 1 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:1: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:2: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:3: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:4: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:5: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:6: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:7: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:28:46.259 INFO:teuthology.orchestra.run.smithi094.stdout:8: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:28:46.260 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2024-09-16T10:28:46.261 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T10:28:46.276 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f51dbe5c-da09-4bb3-8cd3-867d68869245... 2024-09-16T10:28:46.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T10:28:46.319 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T10:28:46.359 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-09-16T10:28:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2024-09-16T10:28:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T10:28:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2024-09-16T10:28:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2024-09-16T10:28:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2024-09-16T10:28:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.360 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2024-09-16T10:28:46.361 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2024-09-16T10:28:46.362 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = "[v1:172.21.15.94:6789],[v1:172.21.15.94:6790],[v1:172.21.15.94:6791],[v1:172.21.15.94:6792],[v1:172.21.15.94:6793],[v1:172.21.15.152:6789],[v1:172.21.15.152:6790],[v1:172.21.15.152:6791],[v1:172.21.15.152:6792]" 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: mon client directed command retry = 5 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: mon election default strategy = 1 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: ms bind msgr2 = False 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: ms inject delay max = 1 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: ms inject delay probability = 0.005 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: ms inject delay type = mon 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: ms inject internal delays = 0.002 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: ms inject socket failures = 2500 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: ms type = async 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:28:46.363 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock profile = high_recovery_ops 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2024-09-16T10:28:46.364 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2024-09-16T10:28:46.365 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: bdev async discard = True 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: bdev enable discard = True 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore allocator = stupid 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore block size = 96636764160 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore fsck on mount = True 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore zero block detection = True 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluefs = 20 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluestore = 20 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-09-16T10:28:46.366 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 20 2024-09-16T10:28:46.367 INFO:teuthology.orchestra.run.smithi094.stdout: debug rocksdb = 10 2024-09-16T10:28:46.367 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T10:28:46.367 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd full ratio = 0.9 2024-09-16T10:28:46.367 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd nearfull ratio = 0.8 2024-09-16T10:28:46.367 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify cached snaps = True 2024-09-16T10:28:46.367 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify missing on start = True 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: osd failsafe full ratio = 0.95 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock override recovery settings = True 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: osd objectstore = bluestore 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: osd op queue = debug_random 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: osd op queue cut off = debug_random 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-09-16T10:28:46.368 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: debug monc = 10 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2024-09-16T10:28:46.369 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2024-09-16T10:28:46.370 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:28:46.370 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2024-09-16T10:28:46.370 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2024-09-16T10:28:46.370 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:28:46.370 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.370 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T10:28:46.370 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: # 1m isn't quite enough 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: mon_down_mkfs_grace = 2m 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_filestore_osds = false 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: mon min osdmap epochs = 25 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune interval = 2 2024-09-16T10:28:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune min = 15 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune txsize = 2 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: mon scrub inject missing keys = 0.05 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: mon scrub interval = 300 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: paxos min = 10 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: paxos service trim min = 5 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: paxos trim min = 10 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T10:28:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: debug monc = 20 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.b] 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.c] 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.d] 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.e] 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.f] 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.g] 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.h] 2024-09-16T10:28:46.374 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.i] 2024-09-16T10:28:46.375 INFO:teuthology.orchestra.run.smithi152.stdout:[global] 2024-09-16T10:28:46.375 INFO:teuthology.orchestra.run.smithi152.stdout: chdir = "" 2024-09-16T10:28:46.375 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T10:28:46.375 INFO:teuthology.orchestra.run.smithi152.stdout: auth supported = cephx 2024-09-16T10:28:46.375 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.375 INFO:teuthology.orchestra.run.smithi152.stdout: filestore xattr use omap = true 2024-09-16T10:28:46.375 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: mon clock drift allowed = 1.000 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: osd crush chooseleaf type = 0 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: auth debug = true 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on old message = true 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on bug = true 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: mon pg warn max object skew = 0 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T10:28:46.376 INFO:teuthology.orchestra.run.smithi152.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default size = 2 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow primary affinity = true 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow pg remap = true 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on legacy crush tunables = false 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on crush straw calc version zero = false 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on no sortbitwise = false 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on osd down out interval zero = false 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on too few osds = false 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T10:28:46.377 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: mon_allow_pool_size_one = true 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: osd default data pool replay window = 5 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: mon allow pool delete = true 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: mon cluster log file level = debug 2024-09-16T10:28:46.378 INFO:teuthology.orchestra.run.smithi152.stdout: debug asserts on shutdown = true 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: mon health detail to clog = false 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: mon host = "[v1:172.21.15.94:6789],[v1:172.21.15.94:6790],[v1:172.21.15.94:6791],[v1:172.21.15.94:6792],[v1:172.21.15.94:6793],[v1:172.21.15.152:6789],[v1:172.21.15.152:6790],[v1:172.21.15.152:6791],[v1:172.21.15.152:6792]" 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: mon client directed command retry = 5 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: mon election default strategy = 1 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: ms bind msgr2 = False 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject delay max = 1 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject delay probability = 0.005 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject delay type = mon 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject internal delays = 0.002 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject socket failures = 2500 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: ms type = async 2024-09-16T10:28:46.379 INFO:teuthology.orchestra.run.smithi152.stdout: fsid = f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout:[osd] 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: osd journal size = 100 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub load threshold = 5.0 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub max interval = 600 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock profile = high_recovery_ops 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: osd recover clone overlap = true 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: osd recovery max chunk = 1048576 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.380 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug shutdown = true 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug op order = true 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify stray on activate = true 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: osd open classes on start = true 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug pg log writeout = true 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: osd deep scrub update digest min age = 30 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: osd map max advance = 10 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: journal zero on create = true 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.381 INFO:teuthology.orchestra.run.smithi152.stdout: filestore ondisk finisher threads = 3 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: filestore apply finisher threads = 3 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: bdev debug aio = true 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug misdirected ops = true 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: bdev async discard = True 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: bdev enable discard = True 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore allocator = stupid 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore block size = 96636764160 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore fsck on mount = True 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore zero block detection = True 2024-09-16T10:28:46.382 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluefs = 20 2024-09-16T10:28:46.383 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluestore = 20 2024-09-16T10:28:46.383 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-16T10:28:46.383 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 20 2024-09-16T10:28:46.383 INFO:teuthology.orchestra.run.smithi152.stdout: debug rocksdb = 10 2024-09-16T10:28:46.383 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T10:28:46.383 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd full ratio = 0.9 2024-09-16T10:28:46.384 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd nearfull ratio = 0.8 2024-09-16T10:28:46.384 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify cached snaps = True 2024-09-16T10:28:46.384 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify missing on start = True 2024-09-16T10:28:46.384 INFO:teuthology.orchestra.run.smithi152.stdout: osd failsafe full ratio = 0.95 2024-09-16T10:28:46.384 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock override recovery settings = True 2024-09-16T10:28:46.384 INFO:teuthology.orchestra.run.smithi152.stdout: osd objectstore = bluestore 2024-09-16T10:28:46.384 INFO:teuthology.orchestra.run.smithi152.stdout: osd op queue = debug_random 2024-09-16T10:28:46.384 INFO:teuthology.orchestra.run.smithi152.stdout: osd op queue cut off = debug_random 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr] 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 20 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: mgr/telemetry/nag = false 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: debug monc = 10 2024-09-16T10:28:46.385 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.386 INFO:teuthology.orchestra.run.smithi152.stdout:[mon] 2024-09-16T10:28:46.386 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-16T10:28:46.386 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-16T10:28:46.386 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 20 2024-09-16T10:28:46.386 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-16T10:28:46.386 INFO:teuthology.orchestra.run.smithi152.stdout: mon data avail warn = 5 2024-09-16T10:28:46.386 INFO:teuthology.orchestra.run.smithi152.stdout: mon mgr mkfs grace = 240 2024-09-16T10:28:46.386 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd reporter subtree level = osd 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd prime pg temp = true 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T10:28:46.387 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: # 1m isn't quite enough 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: mon_down_mkfs_grace = 2m 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_filestore_osds = false 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: mon min osdmap epochs = 25 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: mon osdmap full prune interval = 2 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: mon osdmap full prune min = 15 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: mon osdmap full prune txsize = 2 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: mon scrub inject missing keys = 0.05 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: mon scrub interval = 300 2024-09-16T10:28:46.388 INFO:teuthology.orchestra.run.smithi152.stdout: paxos min = 10 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: paxos service trim min = 5 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: paxos trim min = 10 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout:[client] 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: rgw cache enabled = true 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable ops log = true 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable usage log = true 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: debug monc = 20 2024-09-16T10:28:46.389 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-16T10:28:46.390 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.a] 2024-09-16T10:28:46.390 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.b] 2024-09-16T10:28:46.390 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.c] 2024-09-16T10:28:46.390 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.d] 2024-09-16T10:28:46.390 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.e] 2024-09-16T10:28:46.390 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.f] 2024-09-16T10:28:46.390 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.g] 2024-09-16T10:28:46.390 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.h] 2024-09-16T10:28:46.390 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.i] 2024-09-16T10:28:46.391 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T10:28:46.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T10:28:46.485 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T10:28:46.485 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:46.485 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T10:28:46.500 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:46.500 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T10:28:46.555 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2024-09-16T10:28:46.555 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:46.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T10:28:46.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T10:28:46.629 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:28:46.629 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:28:46.685 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:28:46.685 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:28:46.685 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T10:28:46.685 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T10:28:46.722 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:28:46.723 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:28:46.780 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T10:28:46.780 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T10:28:46.780 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T10:28:46.882 INFO:teuthology.orchestra.run.smithi152.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T10:28:46.885 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T10:28:46.885 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T10:28:46.885 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T10:28:46.926 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T10:28:46.938 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T10:28:46.939 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T10:28:46.939 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T10:28:46.964 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:28:46.964 INFO:tasks.ceph:role: osd.0 2024-09-16T10:28:46.964 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-16T10:28:46.964 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T10:28:47.028 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:28:47.029 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:28:47.029 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:28:47.029 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T10:28:47.029 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:28:47.029 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:28:47.029 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:28:47.029 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T10:28:47.029 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:28:47.029 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:28:47.062 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-09-16T10:28:47.064 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-09-16T10:28:47.064 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T10:28:47.133 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T10:28:47.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T10:28:47.271 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:28:47.271 INFO:tasks.ceph:role: osd.1 2024-09-16T10:28:47.271 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-16T10:28:47.271 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T10:28:47.336 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:28:47.336 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:28:47.336 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:28:47.336 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T10:28:47.337 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:28:47.337 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:28:47.337 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:28:47.337 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T10:28:47.337 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:28:47.337 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:28:47.368 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-09-16T10:28:47.370 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-09-16T10:28:47.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T10:28:47.438 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T10:28:47.510 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T10:28:47.575 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:28:47.575 INFO:tasks.ceph:role: osd.2 2024-09-16T10:28:47.575 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-16T10:28:47.575 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:28:47.639 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:28:47.674 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-09-16T10:28:47.676 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-09-16T10:28:47.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T10:28:47.744 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T10:28:47.813 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:28:47.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:47.905+0000 7ff5fec1f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T10:28:47.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:47.905+0000 7ff5fec1f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T10:28:47.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:47.905+0000 7ff5fec1f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T10:28:47.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:47.905+0000 7ff5fec1f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T10:28:47.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:47.905+0000 7ff5fec1f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T10:28:47.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:47.906+0000 7ff5fec1f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T10:28:50.750 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T10:28:50.778 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:28:50.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:50.865+0000 7fe276554640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T10:28:50.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:50.865+0000 7fe276554640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T10:28:50.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:50.865+0000 7fe276554640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T10:28:50.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:50.865+0000 7fe276554640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T10:28:50.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:50.865+0000 7fe276554640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T10:28:50.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:50.865+0000 7fe276554640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T10:28:53.707 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T10:28:53.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:28:53.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:53.819+0000 7f141281e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T10:28:53.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:53.820+0000 7f141281e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T10:28:53.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:53.820+0000 7f141281e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T10:28:53.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:53.820+0000 7f141281e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T10:28:53.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:53.820+0000 7f141281e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T10:28:53.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:28:53.820+0000 7f141281e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T10:28:56.672 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T10:28:56.700 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T10:28:56.730 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T10:28:56.731 INFO:tasks.ceph:role: osd.3 2024-09-16T10:28:56.731 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:28:56.731 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:28:56.807 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:28:57.321 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-16T10:28:57.323 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-16T10:28:57.323 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T10:28:57.476 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T10:28:57.512 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T10:28:57.580 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T10:28:57.580 INFO:tasks.ceph:role: osd.4 2024-09-16T10:28:57.580 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:28:57.580 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:28:57.654 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:28:58.100 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-16T10:28:58.103 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-16T10:28:58.103 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T10:28:58.206 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T10:28:58.238 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T10:28:58.304 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T10:28:58.305 INFO:tasks.ceph:role: osd.5 2024-09-16T10:28:58.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:28:58.305 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T10:28:58.378 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:28:58.378 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:28:58.378 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:28:58.378 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T10:28:58.379 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:28:58.379 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:28:58.379 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:28:58.379 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T10:28:58.379 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:28:58.379 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:28:58.865 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-16T10:28:58.867 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-16T10:28:58.868 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T10:28:59.021 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T10:28:59.053 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:28:59.150 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:28:59.148+0000 7feb85bc5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T10:28:59.153 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:28:59.150+0000 7feb85bc5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T10:28:59.153 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:28:59.150+0000 7feb85bc5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T10:28:59.153 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:28:59.150+0000 7feb85bc5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T10:28:59.153 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:28:59.150+0000 7feb85bc5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T10:28:59.162 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:28:59.159+0000 7feb85bc5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T10:29:02.025 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T10:29:02.056 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:29:02.147 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:02.144+0000 7f0a08c14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T10:29:02.147 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:02.144+0000 7f0a08c14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T10:29:02.148 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:02.144+0000 7f0a08c14640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T10:29:02.148 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:02.144+0000 7f0a08c14640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T10:29:02.148 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:02.144+0000 7f0a08c14640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T10:29:02.148 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:02.145+0000 7f0a08c14640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T10:29:05.000 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T10:29:05.028 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:29:05.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:05.115+0000 7f24bbe2a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T10:29:05.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:05.115+0000 7f24bbe2a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T10:29:05.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:05.115+0000 7f24bbe2a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T10:29:05.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:05.115+0000 7f24bbe2a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T10:29:05.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:05.115+0000 7f24bbe2a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T10:29:05.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:05.116+0000 7f24bbe2a640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T10:29:07.972 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T10:29:08.001 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T10:29:08.002 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:29:08.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T10:29:08.032 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:29:08.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T10:29:08.100 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:29:08.100 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T10:29:08.168 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:29:08.169 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T10:29:08.197 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:29:08.198 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T10:29:08.266 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:29:08.266 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T10:29:08.334 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:29:08.334 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T10:29:08.401 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:29:08.401 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T10:29:08.458 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T10:29:08.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T10:29:08.461 DEBUG:teuthology.orchestra.run.smithi152:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T10:29:08.502 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2024-09-16T10:29:08.502 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBfCOhmhzUFNhAA4F7txv6egXbT709bts7a2g== 2024-09-16T10:29:08.502 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBiCOhm9E+eMxAArolyqJx0I8+NMmIeQvtvZA== 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBlCOhm6x7rMBAAyitUwxNDTIU7SOyJN4BFhQ== 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBeCOhmIYKMNBAAkW4DvVHBvj0Cdph/bgmbSQ== 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.3] 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBrCOhmpmftCBAAKrbHLIu2orBcfgyn8YhM8w== 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.4] 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBuCOhm2cu3CBAAllp1y5EvTcJ0AvHGuloDmQ== 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.5] 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBxCOhmqhT3BhAAdovEiD5jhPcS1ZOyUCv8Sw== 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-09-16T10:29:08.503 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBeCOhmLxMmNxAAJfK8lplKXmkb+NkrwnIjIQ== 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.0] 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBfCOhmhzUFNhAA4F7txv6egXbT709bts7a2g== 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.1] 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBiCOhm9E+eMxAArolyqJx0I8+NMmIeQvtvZA== 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.2] 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBlCOhm6x7rMBAAyitUwxNDTIU7SOyJN4BFhQ== 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.x] 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBeCOhmIYKMNBAAkW4DvVHBvj0Cdph/bgmbSQ== 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.3] 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBrCOhmpmftCBAAKrbHLIu2orBcfgyn8YhM8w== 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.4] 2024-09-16T10:29:08.523 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBuCOhm2cu3CBAAllp1y5EvTcJ0AvHGuloDmQ== 2024-09-16T10:29:08.524 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.5] 2024-09-16T10:29:08.524 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBxCOhmqhT3BhAAdovEiD5jhPcS1ZOyUCv8Sw== 2024-09-16T10:29:08.524 INFO:teuthology.orchestra.run.smithi152.stdout:[client.0] 2024-09-16T10:29:08.524 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBeCOhmLxMmNxAAJfK8lplKXmkb+NkrwnIjIQ== 2024-09-16T10:29:08.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.547 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.620 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.672 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.674 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T10:29:08.726 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T10:29:08.777 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.779 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.830 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.833 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.883 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.886 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:29:08.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T10:29:08.937 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T10:29:08.990 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T10:29:08.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T10:29:09.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:29:09.110 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T10:29:09.135 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T10:29:09.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:29:09.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T10:29:09.321 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T10:29:09.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:29:09.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T10:29:09.507 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2024-09-16T10:29:09.572 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:29:09.666 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2024-09-16T10:29:09.691 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2024-09-16T10:29:09.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:29:09.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2024-09-16T10:29:09.877 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2024-09-16T10:29:09.907 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:29:10.009 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2024-09-16T10:29:10.035 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2024-09-16T10:29:10.101 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:29:10.198 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2024-09-16T10:29:10.222 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2024-09-16T10:29:10.287 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:29:10.381 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2024-09-16T10:29:10.406 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2024-09-16T10:29:10.471 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:29:10.568 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2024-09-16T10:29:10.593 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:29:10.595 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:29:10.648 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T10:29:10.649 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T10:29:10.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T10:29:10.653 INFO:tasks.ceph.mon.a:Started 2024-09-16T10:29:10.653 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T10:29:10.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T10:29:10.655 INFO:tasks.ceph.mon.b:Started 2024-09-16T10:29:10.655 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T10:29:10.655 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T10:29:10.657 INFO:tasks.ceph.mon.c:Started 2024-09-16T10:29:10.658 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-16T10:29:10.658 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-16T10:29:10.660 INFO:tasks.ceph.mon.d:Started 2024-09-16T10:29:10.660 INFO:tasks.ceph.mon.e:Restarting daemon 2024-09-16T10:29:10.660 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-09-16T10:29:10.663 INFO:tasks.ceph.mon.e:Started 2024-09-16T10:29:10.663 INFO:tasks.ceph.mon.f:Restarting daemon 2024-09-16T10:29:10.663 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-09-16T10:29:10.691 INFO:tasks.ceph.mon.f:Started 2024-09-16T10:29:10.691 INFO:tasks.ceph.mon.g:Restarting daemon 2024-09-16T10:29:10.691 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-09-16T10:29:10.693 INFO:tasks.ceph.mon.g:Started 2024-09-16T10:29:10.693 INFO:tasks.ceph.mon.h:Restarting daemon 2024-09-16T10:29:10.693 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-09-16T10:29:10.695 INFO:tasks.ceph.mon.h:Started 2024-09-16T10:29:10.695 INFO:tasks.ceph.mon.i:Restarting daemon 2024-09-16T10:29:10.695 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-09-16T10:29:10.697 INFO:tasks.ceph.mon.i:Started 2024-09-16T10:29:10.698 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T10:29:10.698 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T10:29:10.698 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T10:29:10.700 INFO:tasks.ceph.mgr.x:Started 2024-09-16T10:29:10.700 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T10:29:10.700 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T10:29:10.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.815+0000 7fb6c0b45640 10 monclient: get_monmap_and_config 2024-09-16T10:29:10.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.815+0000 7fb6c0b45640 10 monclient: build_initial_monmap 2024-09-16T10:29:10.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.815+0000 7fb6c0b45640 10 monclient: monmap: 2024-09-16T10:29:10.817 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:10.817 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:10.817 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:10.816019+0000 2024-09-16T10:29:10.817 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:10.816019+0000 2024-09-16T10:29:10.817 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:10.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.815+0000 7fb6c0b45640 1 Processor -- start 2024-09-16T10:29:10.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.815+0000 7fb6c0b45640 1 -- start start 2024-09-16T10:29:10.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.815+0000 7fb6c0b45640 10 monclient: init 2024-09-16T10:29:10.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:10.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 10 monclient: _add_conns ranks=[1,6,2,8,5,3,4,7,0] 2024-09-16T10:29:10.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 10 monclient(hunting): picked mon.noname-f con 0x7fb6bc0a38d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:10.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 10 monclient(hunting): picked mon.noname-d con 0x7fb6bc06c670 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:10.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 10 monclient(hunting): picked mon.noname-b con 0x7fb6bc0a43c0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:10.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc061ae0 con 0x7fb6bc0a38d0 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc0a04d0 con 0x7fb6bc0a43c0 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc0a3250 con 0x7fb6bc06c670 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6c0b45640 10 monclient(hunting): authenticate will time out at 718.156502s 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6bad9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fb6bc0a43c0 0x7fb6bc13f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41944/0 (socket says 172.21.15.94:41944) 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6b9d9b640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fb6bc06c670 0x7fb6bc13bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:48120/0 (socket says 172.21.15.94:48120) 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.816+0000 7fb6bad9d640 1 -- 172.21.15.94:0/1589142896 learned_addr learned my addr 172.21.15.94:0/1589142896 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.818+0000 7fb6bad9d640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.94:6790/0 conn(0x7fb6bc0a43c0 legacy=0x7fb6bc13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.818+0000 7fb6b9d9b640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.94:6792/0 conn(0x7fb6bc06c670 legacy=0x7fb6bc13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:10.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:10.818+0000 7fb6ba59c640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.152:6789/0 conn(0x7fb6bc0a38d0 legacy=0x7fb6bc0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:13.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 10 monclient(hunting): tick 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 1 monclient(hunting): continuing hunt 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.94:6792/0 conn(0x7fb6bc06c670 legacy=0x7fb6bc13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.94:6790/0 conn(0x7fb6bc0a43c0 legacy=0x7fb6bc13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.152:6789/0 conn(0x7fb6bc0a38d0 legacy=0x7fb6bc0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 10 monclient: _add_conns ranks=[7,8,0,3,4,5] 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 10 monclient(hunting): picked mon.noname-i con 0x7fb69c003960 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 10 monclient(hunting): picked mon.noname-e con 0x7fb69c0075f0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 10 monclient(hunting): picked mon.noname-a con 0x7fb69c00b380 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 1 -- 172.21.15.94:0/1589142896 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc0a0a00 con 0x7fb69c00b380 2024-09-16T10:29:13.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 1 -- 172.21.15.94:0/1589142896 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c010190 con 0x7fb69c003960 2024-09-16T10:29:13.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 1 -- 172.21.15.94:0/1589142896 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0113c0 con 0x7fb69c0075f0 2024-09-16T10:29:13.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.815+0000 7fb6abfff640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:13.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.818+0000 7fb6bad9d640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.94:6789/0 conn(0x7fb69c00b380 legacy=0x7fb69c00d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:13.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.818+0000 7fb6b9d9b640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.94:6793/0 conn(0x7fb69c0075f0 legacy=0x7fb69c009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:13.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:13.818+0000 7fb6ba59c640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.152:6792/0 conn(0x7fb69c003960 legacy=0x7fb69c005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777227499 0 0) 0x7fb6bc0a0a00 con 0x7fb69c00b380 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): my global_id is 4104 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a4003520 con 0x7fb69c00b380 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91405943 0 0) 0x7fb69c010190 con 0x7fb69c003960 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:16.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): my global_id is 4120 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6bc0a0a00 con 0x7fb69c003960 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175488173 0 0) 0x7fb69c0113c0 con 0x7fb69c0075f0 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient(hunting): my global_id is 4112 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c010190 con 0x7fb69c0075f0 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1434324179 0 0) 0x7fb69c010190 con 0x7fb69c0075f0 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.152:6792/0 conn(0x7fb69c003960 legacy=0x7fb69c005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.94:6789/0 conn(0x7fb69c00b380 legacy=0x7fb69c00d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:16.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: _finish_hunting 0 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 monclient: found mon.noname-e 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6bad9d640 1 -- 172.21.15.94:0/1589142896 reap_dead start 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c0125f0 con 0x7fb69c0075f0 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: _finish_auth 0 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: _check_auth_tickets 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb6b0001d10 con 0x7fb69c0075f0 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: dump: 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:16.455 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6c0b45640 5 monclient: authenticate success, global_id 4112 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6c0b45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6b0001fa0 con 0x7fb69c0075f0 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 1 -- 172.21.15.94:0/1589142896 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb6b0004ea0 con 0x7fb69c0075f0 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.448+0000 7fb6b959a640 10 monclient: dump: 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:16.456 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:16.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.449+0000 7fb6c0b45640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:16.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.449+0000 7fb6c0b45640 4 set_mon_vals no callback set 2024-09-16T10:29:16.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.449+0000 7fb6c0b45640 10 monclient: shutdown 2024-09-16T10:29:16.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.450+0000 7fb6c0b45640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:16.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.450+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1589142896 >> v1:172.21.15.94:6793/0 conn(0x7fb69c0075f0 legacy=0x7fb69c009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:16.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.450+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1589142896 shutdown_connections 2024-09-16T10:29:16.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.450+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1589142896 >> 172.21.15.94:0/1589142896 conn(0x7fb6bc0a59d0 msgr2=0x7fb6bc0a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.450+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1589142896 shutdown_connections 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.450+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1589142896 wait complete. 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.451+0000 7fb6c0b45640 10 monclient: build_initial_monmap 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.451+0000 7fb6c0b45640 10 monclient: monmap: 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:16.463 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.452+0000 7fb6c0b45640 1 Processor -- start 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.452+0000 7fb6c0b45640 1 -- start start 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.452+0000 7fb6c0b45640 10 monclient: init 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 10 monclient: _add_conns ranks=[8,6,0,1,4,7,5,3,2] 2024-09-16T10:29:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 10 monclient(hunting): picked mon.noname-e con 0x7fb69c0075f0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 10 monclient(hunting): picked mon.noname-d con 0x7fb6bc137340 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 10 monclient(hunting): picked mon.noname-a con 0x7fb6bc1caca0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc09f3c0 con 0x7fb6bc1caca0 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc1257d0 con 0x7fb6bc137340 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc09f090 con 0x7fb69c0075f0 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6c0b45640 10 monclient(hunting): authenticate will time out at 723.794198s 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6ba59c640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7fb69c0075f0 0x7fb6bc136c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:45244/0 (socket says 172.21.15.94:45244) 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.453+0000 7fb6ba59c640 1 -- 172.21.15.94:0/1410447779 learned_addr learned my addr 172.21.15.94:0/1410447779 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6ba59c640 1 -- 172.21.15.94:0/1410447779 >> v1:172.21.15.94:6793/0 conn(0x7fb69c0075f0 legacy=0x7fb6bc136c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:16.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6bad9d640 1 -- 172.21.15.94:0/1410447779 >> v1:172.21.15.94:6789/0 conn(0x7fb6bc1caca0 legacy=0x7fb6bc1d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272247414 0 0) 0x7fb6bc09f3c0 con 0x7fb6bc1caca0 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 10 monclient(hunting): my global_id is 4113 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c0091b0 con 0x7fb6bc1caca0 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3975839993 0 0) 0x7fb69c0091b0 con 0x7fb6bc1caca0 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6bc09f3c0 con 0x7fb6bc1caca0 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb6ac001c00 con 0x7fb6bc1caca0 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 10 monclient(hunting): dump: 2024-09-16T10:29:16.466 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:16.467 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.455+0000 7fb6aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1977029080 0 0) 0x7fb6bc09f3c0 con 0x7fb6bc1caca0 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 >> v1:172.21.15.94:6793/0 conn(0x7fb69c0075f0 legacy=0x7fb6bc136c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 >> v1:172.21.15.94:6792/0 conn(0x7fb6bc137340 legacy=0x7fb6bc1d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 1 monclient: found mon.a 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc127340 con 0x7fb6bc1caca0 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6c0b45640 5 monclient: authenticate success, global_id 4113 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6c0b45640 10 monclient: _renew_subs 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6c0b45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1410447779 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc127620 con 0x7fb6bc1caca0 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6c0b45640 10 monclient: _renew_subs 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6c0b45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1410447779 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc1252c0 con 0x7fb6bc1caca0 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6ac004a90 con 0x7fb6bc1caca0 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb6ac004f40 con 0x7fb6bc1caca0 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:16.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6b8d99640 4 set_mon_vals no callback set 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 10 monclient: dump: 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fb6ac0053a0 con 0x7fb6bc1caca0 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.456+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 776910775 0 0) 0x7fb6ac006570 con 0x7fb6bc1caca0 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.457+0000 7fb6c0b45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.457+0000 7fb6c0b45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.457+0000 7fb6c0b45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.457+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1410447779 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a0005040 con 0x7fb6bc1caca0 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.459+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fb6ac0058f0 con 0x7fb6bc1caca0 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.459+0000 7fb6aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.459+0000 7fb6aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:16.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.521+0000 7fb6c0b45640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T10:29:16.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.521+0000 7fb6c0b45640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T10:29:16.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.521+0000 7fb6c0b45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:16.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:16.521+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1410447779 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb6a00052f0 con 0x7fb6bc1caca0 2024-09-16T10:29:17.094 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:17.092+0000 7fb443923040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T10:29:17.135 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:17.133+0000 7fb443923040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T10:29:17.275 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:17.274+0000 7fb443923040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T10:29:17.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.298+0000 7fb6aaffd640 1 -- 172.21.15.94:0/1410447779 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fb6ac005bf0 con 0x7fb6bc1caca0 2024-09-16T10:29:17.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.298+0000 7fb6aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T10:29:17.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.298+0000 7fb6aaffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T10:29:17.301 INFO:teuthology.orchestra.run.smithi094.stderr:adjusted tunables profile to default 2024-09-16T10:29:17.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.299+0000 7fb6c0b45640 10 monclient: shutdown 2024-09-16T10:29:17.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.299+0000 7fb6c0b45640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:17.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.299+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1410447779 >> v1:172.21.15.94:6789/0 conn(0x7fb6bc1caca0 legacy=0x7fb6bc1d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:17.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.299+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1410447779 shutdown_connections 2024-09-16T10:29:17.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.299+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1410447779 >> 172.21.15.94:0/1410447779 conn(0x7fb6bc0a59d0 msgr2=0x7fb6bc0a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:17.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.299+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1410447779 shutdown_connections 2024-09-16T10:29:17.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:17.299+0000 7fb6c0b45640 1 -- 172.21.15.94:0/1410447779 wait complete. 2024-09-16T10:29:17.314 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T10:29:17.314 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T10:29:17.315 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:29:17.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T10:29:17.326 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:17.325+0000 7fb443923040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T10:29:17.342 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:29:17.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T10:29:17.409 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-16T10:29:17.409 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T10:29:17.476 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:29:17.476 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T10:29:17.489 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:17.488+0000 7fb443923040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T10:29:17.501 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:29:17.501 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T10:29:17.566 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T10:29:17.566 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T10:29:17.602 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:17.601+0000 7fb443923040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T10:29:17.627 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 19cab4f7-c271-4ce8-895e-75a5871502f0 0 2024-09-16T10:29:17.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.761+0000 7fe4da420640 10 monclient: get_monmap_and_config 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.761+0000 7fe4da420640 10 monclient: build_initial_monmap 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.761+0000 7fe4da420640 10 monclient: monmap: 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:29:17.761797+0000 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:29:17.761797+0000 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:17.763 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:17.764 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:17.764 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:17.764 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:17.764 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:17.764 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:17.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.761+0000 7fe4da420640 1 Processor -- start 2024-09-16T10:29:17.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.761+0000 7fe4da420640 1 -- start start 2024-09-16T10:29:17.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.761+0000 7fe4da420640 10 monclient: init 2024-09-16T10:29:17.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:17.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 10 monclient: _add_conns ranks=[4,8,5,0,6,7,2,3,1] 2024-09-16T10:29:17.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d40a2140 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:17.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 10 monclient(hunting): picked mon.noname-e con 0x7fe4d40a15e0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:17.766 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 10 monclient(hunting): picked mon.noname-h con 0x7fe4d40a19f0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:17.766 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d4061ae0 con 0x7fe4d40a2140 2024-09-16T10:29:17.766 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40a0920 con 0x7fe4d40a19f0 2024-09-16T10:29:17.766 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40a0360 con 0x7fe4d40a15e0 2024-09-16T10:29:17.766 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:17.766 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4da420640 10 monclient(hunting): authenticate will time out at 713.419312s 2024-09-16T10:29:17.766 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4d89bd640 1 --1- >> v1:172.21.15.152:6791/0 conn(0x7fe4d40a19f0 0x7fe4d413ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6791/0 says I am v1:172.21.15.152:37678/0 (socket says 172.21.15.152:37678) 2024-09-16T10:29:17.766 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.762+0000 7fe4d89bd640 1 -- 172.21.15.152:0/3107741536 learned_addr learned my addr 172.21.15.152:0/3107741536 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:17.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.764+0000 7fe4d89bd640 1 -- 172.21.15.152:0/3107741536 >> v1:172.21.15.152:6791/0 conn(0x7fe4d40a19f0 legacy=0x7fe4d413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:17.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.764+0000 7fe4d37fe640 1 -- 172.21.15.152:0/3107741536 >> v1:172.21.15.94:6793/0 conn(0x7fe4d40a15e0 legacy=0x7fe4d413b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.764+0000 7fe4d3fff640 1 -- 172.21.15.152:0/3107741536 >> v1:172.21.15.94:6791/0 conn(0x7fe4d40a2140 legacy=0x7fe4d40a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.764+0000 7fe4d2ffd640 1 -- 172.21.15.152:0/3107741536 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31041287 0 0) 0x7fe4d40a0360 con 0x7fe4d40a15e0 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient(hunting): my global_id is 4121 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 1 -- 172.21.15.152:0/3107741536 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c0003520 con 0x7fe4d40a15e0 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 1 -- 172.21.15.152:0/3107741536 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3952539258 0 0) 0x7fe4c0003520 con 0x7fe4d40a15e0 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 1 -- 172.21.15.152:0/3107741536 >> v1:172.21.15.152:6791/0 conn(0x7fe4d40a19f0 legacy=0x7fe4d413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 1 -- 172.21.15.152:0/3107741536 >> v1:172.21.15.94:6791/0 conn(0x7fe4d40a2140 legacy=0x7fe4d40a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 1 monclient: found mon.noname-e 2024-09-16T10:29:17.770 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 1 -- 172.21.15.152:0/3107741536 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d40a0640 con 0x7fe4d40a15e0 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 1 -- 172.21.15.152:0/3107741536 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4bc001b70 con 0x7fe4d40a15e0 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4da420640 5 monclient: authenticate success, global_id 4121 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4da420640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: dump: 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:17.771 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:17.772 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4da420640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 1 -- 172.21.15.152:0/3107741536 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4bc003610 con 0x7fe4d40a15e0 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4da420640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 1 -- 172.21.15.152:0/3107741536 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4bc003ac0 con 0x7fe4d40a15e0 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4da420640 4 set_mon_vals no callback set 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.765+0000 7fe4d2ffd640 10 monclient: dump: 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:17.773 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:17.774 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.766+0000 7fe4da420640 10 monclient: shutdown 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.766+0000 7fe4da420640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.766+0000 7fe4da420640 1 -- 172.21.15.152:0/3107741536 >> v1:172.21.15.94:6793/0 conn(0x7fe4d40a15e0 legacy=0x7fe4d413b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.766+0000 7fe4da420640 1 -- 172.21.15.152:0/3107741536 shutdown_connections 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.766+0000 7fe4da420640 1 -- 172.21.15.152:0/3107741536 >> 172.21.15.152:0/3107741536 conn(0x7fe4d40a93c0 msgr2=0x7fe4d40a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.766+0000 7fe4da420640 1 -- 172.21.15.152:0/3107741536 shutdown_connections 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.766+0000 7fe4da420640 1 -- 172.21.15.152:0/3107741536 wait complete. 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.766+0000 7fe4da420640 10 monclient: build_initial_monmap 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.766+0000 7fe4da420640 10 monclient: monmap: 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-09-16T10:29:17.775 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 1 Processor -- start 2024-09-16T10:29:17.776 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 1 -- start start 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 10 monclient: init 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 10 monclient: _add_conns ranks=[7,2,1,5,3,0,8,6,4] 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 10 monclient(hunting): picked mon.noname-i con 0x7fe4d40a15e0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d40a19f0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 10 monclient(hunting): picked mon.noname-f con 0x7fe4d40a2140 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40a7ca0 con 0x7fe4d40a2140 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d4139790 con 0x7fe4d40a19f0 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d409f8e0 con 0x7fe4d40a15e0 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4da420640 10 monclient(hunting): authenticate will time out at 713.424832s 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4d3fff640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7fe4d40a15e0 0x7fe4d413ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.152:52538/0 (socket says 172.21.15.152:52538) 2024-09-16T10:29:17.777 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.767+0000 7fe4d3fff640 1 -- 172.21.15.152:0/684897397 learned_addr learned my addr 172.21.15.152:0/684897397 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:17.778 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.770+0000 7fe4d89bd640 1 -- 172.21.15.152:0/684897397 >> v1:172.21.15.152:6789/0 conn(0x7fe4d40a2140 legacy=0x7fe4d41d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:17.778 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.770+0000 7fe4d3fff640 1 -- 172.21.15.152:0/684897397 >> v1:172.21.15.152:6792/0 conn(0x7fe4d40a15e0 legacy=0x7fe4d413ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:17.778 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.770+0000 7fe4d37fe640 1 -- 172.21.15.152:0/684897397 >> v1:172.21.15.94:6790/0 conn(0x7fe4d40a19f0 legacy=0x7fe4d413f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:17.918 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:17.916+0000 7fb443923040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T10:29:17.972 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:17.971+0000 7fb443923040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.991+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754607949 0 0) 0x7fe4d4139790 con 0x7fe4d40a19f0 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.991+0000 7fe4d0ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.991+0000 7fe4d0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.991+0000 7fe4d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.991+0000 7fe4d0ff9640 10 monclient(hunting): my global_id is 4115 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.991+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b4003520 con 0x7fe4d40a19f0 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.991+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1312113743 0 0) 0x7fe4b4003520 con 0x7fe4d40a19f0 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.991+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d4139790 con 0x7fe4d40a19f0 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.991+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4bc001b30 con 0x7fe4d40a19f0 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient(hunting): dump: 2024-09-16T10:29:17.995 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:17.996 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 997731094 0 0) 0x7fe4d4139790 con 0x7fe4d40a19f0 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 >> v1:172.21.15.152:6792/0 conn(0x7fe4d40a15e0 legacy=0x7fe4d413ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 >> v1:172.21.15.152:6789/0 conn(0x7fe4d40a2140 legacy=0x7fe4d41d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient: _finish_hunting 0 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 1 monclient: found mon.b 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d40a76e0 con 0x7fe4d40a19f0 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient: _finish_auth 0 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:17.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4da420640 5 monclient: authenticate success, global_id 4115 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4da420640 10 monclient: _renew_subs 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4da420640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4da420640 1 -- 172.21.15.152:0/684897397 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d4121950 con 0x7fe4d40a19f0 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4da420640 10 monclient: _renew_subs 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4da420640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4da420640 1 -- 172.21.15.152:0/684897397 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d411f580 con 0x7fe4d40a19f0 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4bc0040a0 con 0x7fe4d40a19f0 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4bc0057c0 con 0x7fe4d40a19f0 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.992+0000 7fe4d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.993+0000 7fe4d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:17.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.993+0000 7fe4d0ff9640 10 monclient: dump: 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:17.999 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:18.000 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:18.000 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:18.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.993+0000 7fe4d27fc640 4 set_mon_vals no callback set 2024-09-16T10:29:18.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.993+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fe4bc005c20 con 0x7fe4d40a19f0 2024-09-16T10:29:18.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.993+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1340796961 0 0) 0x7fe4bc006bc0 con 0x7fe4d40a19f0 2024-09-16T10:29:18.000 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.993+0000 7fe4da420640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:18.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.994+0000 7fe4da420640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:18.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.994+0000 7fe4da420640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:18.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.994+0000 7fe4da420640 1 -- 172.21.15.152:0/684897397 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b8005040 con 0x7fe4d40a19f0 2024-09-16T10:29:18.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.996+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fe4bc005ed0 con 0x7fe4d40a19f0 2024-09-16T10:29:18.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.996+0000 7fe4d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:18.001 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:17.996+0000 7fe4d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:18.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:18.064+0000 7fe4da420640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "19cab4f7-c271-4ce8-895e-75a5871502f0", "id": 0}] 2024-09-16T10:29:18.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:18.064+0000 7fe4da420640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "19cab4f7-c271-4ce8-895e-75a5871502f0", "id": 0}] 2024-09-16T10:29:18.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:18.064+0000 7fe4da420640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:18.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:18.064+0000 7fe4da420640 1 -- 172.21.15.152:0/684897397 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd new", "uuid": "19cab4f7-c271-4ce8-895e-75a5871502f0", "id": 0} v 0) v1 -- 0x7fe4b8005870 con 0x7fe4d40a19f0 2024-09-16T10:29:18.124 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:18.122+0000 7fb443923040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T10:29:19.075 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:19.073+0000 7fb443923040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T10:29:19.125 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:19.123+0000 7fb443923040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T10:29:19.209 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.207+0000 7fe4d0ff9640 1 -- 172.21.15.152:0/684897397 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "19cab4f7-c271-4ce8-895e-75a5871502f0", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3379741570 0 732707515) 0x7fe4bc0061c0 con 0x7fe4d40a19f0 2024-09-16T10:29:19.209 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.207+0000 7fe4d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "19cab4f7-c271-4ce8-895e-75a5871502f0", "id": 0}] 2024-09-16T10:29:19.209 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.207+0000 7fe4d0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:19.209 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-09-16T10:29:19.210 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.208+0000 7fe4da420640 10 monclient: shutdown 2024-09-16T10:29:19.210 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.208+0000 7fe4da420640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:19.210 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.208+0000 7fe4da420640 1 -- 172.21.15.152:0/684897397 >> v1:172.21.15.94:6790/0 conn(0x7fe4d40a19f0 legacy=0x7fe4d413f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.210 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.208+0000 7fe4da420640 1 -- 172.21.15.152:0/684897397 shutdown_connections 2024-09-16T10:29:19.210 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.208+0000 7fe4da420640 1 -- 172.21.15.152:0/684897397 >> 172.21.15.152:0/684897397 conn(0x7fe4d40a93c0 msgr2=0x7fe4d4073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:19.210 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.208+0000 7fe4da420640 1 -- 172.21.15.152:0/684897397 shutdown_connections 2024-09-16T10:29:19.210 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.208+0000 7fe4da420640 1 -- 172.21.15.152:0/684897397 wait complete. 2024-09-16T10:29:19.220 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 6641fac5-e871-4401-bf30-598b633e70b4 1 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.314+0000 7fe7389a0640 10 monclient: get_monmap_and_config 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.314+0000 7fe7389a0640 10 monclient: build_initial_monmap 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.314+0000 7fe7389a0640 10 monclient: monmap: 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:29:19.315007+0000 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:29:19.315007+0000 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:19.317 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:19.317 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:19.317 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:19.317 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:19.317 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:19.317 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:19.317 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.314+0000 7fe7389a0640 1 Processor -- start 2024-09-16T10:29:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 1 -- start start 2024-09-16T10:29:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 10 monclient: init 2024-09-16T10:29:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 10 monclient: _add_conns ranks=[0,8,3,1,2,5,6,4,7] 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 10 monclient(hunting): picked mon.noname-a con 0x7fe7340714a0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 10 monclient(hunting): picked mon.noname-e con 0x7fe7340a28d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 10 monclient(hunting): picked mon.noname-g con 0x7fe7340a2d30 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73404f100 con 0x7fe7340714a0 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe734136e90 con 0x7fe7340a2d30 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe734071880 con 0x7fe7340a28d0 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe7389a0640 10 monclient(hunting): authenticate will time out at 714.972549s 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe733fff640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7fe7340a2d30 0x7fe73413ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.152:44100/0 (socket says 172.21.15.152:44100) 2024-09-16T10:29:19.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.315+0000 7fe733fff640 1 -- 172.21.15.152:0/1727604234 learned_addr learned my addr 172.21.15.152:0/1727604234 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.317+0000 7fe733fff640 1 -- 172.21.15.152:0/1727604234 >> v1:172.21.15.152:6790/0 conn(0x7fe7340a2d30 legacy=0x7fe73413ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.317+0000 7fe7327fc640 1 -- 172.21.15.152:0/1727604234 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210936110 0 0) 0x7fe72c0017d0 con 0x7fe7340a2d30 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.317+0000 7fe7327fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.317+0000 7fe7327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.317+0000 7fe7327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.317+0000 7fe7327fc640 10 monclient(hunting): my global_id is 4116 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.317+0000 7fe7327fc640 1 -- 172.21.15.152:0/1727604234 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c003520 con 0x7fe7340a2d30 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7337fe640 1 -- 172.21.15.152:0/1727604234 >> v1:172.21.15.94:6789/0 conn(0x7fe7340714a0 legacy=0x7fe734135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe732ffd640 1 -- 172.21.15.152:0/1727604234 >> v1:172.21.15.94:6793/0 conn(0x7fe7340a28d0 legacy=0x7fe73413b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 1 -- 172.21.15.152:0/1727604234 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063932310 0 0) 0x7fe71c003520 con 0x7fe7340a2d30 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 1 -- 172.21.15.152:0/1727604234 >> v1:172.21.15.94:6793/0 conn(0x7fe7340a28d0 legacy=0x7fe73413b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.323 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 1 -- 172.21.15.152:0/1727604234 >> v1:172.21.15.94:6789/0 conn(0x7fe7340714a0 legacy=0x7fe734135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 1 monclient: found mon.noname-g 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.152:6790/0 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 1 -- 172.21.15.152:0/1727604234 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73409ada0 con 0x7fe7340a2d30 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 1 -- 172.21.15.152:0/1727604234 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe72c001f70 con 0x7fe7340a2d30 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 5 monclient: authenticate success, global_id 4116 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: dump: 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.325 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 1 -- 172.21.15.152:0/1727604234 <== mon.3 v1:172.21.15.152:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe72c004c60 con 0x7fe7340a2d30 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 1 -- 172.21.15.152:0/1727604234 <== mon.3 v1:172.21.15.152:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe72c005150 con 0x7fe7340a2d30 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 4 set_mon_vals no callback set 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7327fc640 10 monclient: dump: 2024-09-16T10:29:19.326 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.327 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 10 monclient: shutdown 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 1 -- 172.21.15.152:0/1727604234 >> v1:172.21.15.152:6790/0 conn(0x7fe7340a2d30 legacy=0x7fe73413ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 1 -- 172.21.15.152:0/1727604234 shutdown_connections 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 1 -- 172.21.15.152:0/1727604234 >> 172.21.15.152:0/1727604234 conn(0x7fe7340a5cc0 msgr2=0x7fe734065c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 1 -- 172.21.15.152:0/1727604234 shutdown_connections 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.318+0000 7fe7389a0640 1 -- 172.21.15.152:0/1727604234 wait complete. 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.319+0000 7fe7389a0640 10 monclient: build_initial_monmap 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.319+0000 7fe7389a0640 10 monclient: monmap: 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:19.328 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:19.329 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.319+0000 7fe7389a0640 1 Processor -- start 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 1 -- start start 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 10 monclient: init 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 10 monclient: _add_conns ranks=[2,5,8,0,6,4,7,3,1] 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 10 monclient(hunting): picked mon.noname-b con 0x7fe7340714a0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 10 monclient(hunting): picked mon.noname-h con 0x7fe7340a28d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 10 monclient(hunting): picked mon.noname-e con 0x7fe7340a2d30 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7340a7ea0 con 0x7fe7340714a0 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73412ca80 con 0x7fe7340a28d0 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73412b950 con 0x7fe7340a2d30 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe7389a0640 10 monclient(hunting): authenticate will time out at 714.977522s 2024-09-16T10:29:19.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe732ffd640 1 --1- >> v1:172.21.15.152:6791/0 conn(0x7fe7340a28d0 0x7fe73412b240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6791/0 says I am v1:172.21.15.152:37680/0 (socket says 172.21.15.152:37680) 2024-09-16T10:29:19.331 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.320+0000 7fe732ffd640 1 -- 172.21.15.152:0/1918010697 learned_addr learned my addr 172.21.15.152:0/1918010697 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:19.337 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.322+0000 7fe732ffd640 1 -- 172.21.15.152:0/1918010697 >> v1:172.21.15.152:6791/0 conn(0x7fe7340a28d0 legacy=0x7fe73412b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.337 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.322+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447171245 0 0) 0x7fe73412ca80 con 0x7fe7340a28d0 2024-09-16T10:29:19.337 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.322+0000 7fe71bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.322+0000 7fe71bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.322+0000 7fe71bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.322+0000 7fe71bfff640 10 monclient(hunting): my global_id is 4118 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.322+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe720003520 con 0x7fe7340a28d0 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe733fff640 1 -- 172.21.15.152:0/1918010697 >> v1:172.21.15.94:6793/0 conn(0x7fe7340a2d30 legacy=0x7fe7341d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe7337fe640 1 -- 172.21.15.152:0/1918010697 >> v1:172.21.15.94:6790/0 conn(0x7fe7340714a0 legacy=0x7fe73412ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3995171381 0 0) 0x7fe720003520 con 0x7fe7340a28d0 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe73412ca80 con 0x7fe7340a28d0 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe728001d20 con 0x7fe7340a28d0 2024-09-16T10:29:19.338 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): dump: 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357453041 0 0) 0x7fe7340a7ea0 con 0x7fe7340714a0 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): my global_id is 4124 2024-09-16T10:29:19.340 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe720003520 con 0x7fe7340714a0 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512233938 0 0) 0x7fe73412b950 con 0x7fe7340a2d30 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient(hunting): my global_id is 4139 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7340a7ea0 con 0x7fe7340a2d30 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1579902653 0 0) 0x7fe73412ca80 con 0x7fe7340a28d0 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 >> v1:172.21.15.94:6793/0 conn(0x7fe7340a2d30 legacy=0x7fe7341d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 >> v1:172.21.15.94:6790/0 conn(0x7fe7340714a0 legacy=0x7fe73412ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 monclient: found mon.h 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:19.341 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 --> v1:172.21.15.152:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7340a82d0 con 0x7fe7340a28d0 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient: _finish_auth 0 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe7389a0640 5 monclient: authenticate success, global_id 4118 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe7389a0640 10 monclient: _renew_subs 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe7389a0640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe7389a0640 1 -- 172.21.15.152:0/1918010697 --> v1:172.21.15.152:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73412e8c0 con 0x7fe7340a28d0 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe7389a0640 10 monclient: _renew_subs 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe7389a0640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe7389a0640 1 -- 172.21.15.152:0/1918010697 --> v1:172.21.15.152:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73412d350 con 0x7fe7340a28d0 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe728003b30 con 0x7fe7340a28d0 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe728003fe0 con 0x7fe7340a28d0 2024-09-16T10:29:19.342 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe731ffb640 4 set_mon_vals no callback set 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 10 monclient: dump: 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.343 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fe7280057c0 con 0x7fe7340a28d0 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.323+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3034619437 0 0) 0x7fe7280069e0 con 0x7fe7340a28d0 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.324+0000 7fe7389a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.324+0000 7fe7389a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.324+0000 7fe7389a0640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.324+0000 7fe7389a0640 1 -- 172.21.15.152:0/1918010697 --> v1:172.21.15.152:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe714005040 con 0x7fe7340a28d0 2024-09-16T10:29:19.344 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.328+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fe728005a70 con 0x7fe7340a28d0 2024-09-16T10:29:19.345 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.328+0000 7fe71bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.345 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.328+0000 7fe71bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:19.398 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.396+0000 7fe7389a0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6641fac5-e871-4401-bf30-598b633e70b4", "id": 1}] 2024-09-16T10:29:19.398 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.396+0000 7fe7389a0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6641fac5-e871-4401-bf30-598b633e70b4", "id": 1}] 2024-09-16T10:29:19.398 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.396+0000 7fe7389a0640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:19.398 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.396+0000 7fe7389a0640 1 -- 172.21.15.152:0/1918010697 --> v1:172.21.15.152:6791/0 -- mon_command({"prefix": "osd new", "uuid": "6641fac5-e871-4401-bf30-598b633e70b4", "id": 1} v 0) v1 -- 0x7fe714005870 con 0x7fe7340a28d0 2024-09-16T10:29:19.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.400+0000 7fe71bfff640 1 -- 172.21.15.152:0/1918010697 <== mon.5 v1:172.21.15.152:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6641fac5-e871-4401-bf30-598b633e70b4", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 693014455 0 940483276) 0x7fe728005c90 con 0x7fe7340a28d0 2024-09-16T10:29:19.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.400+0000 7fe71bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6641fac5-e871-4401-bf30-598b633e70b4", "id": 1}] 2024-09-16T10:29:19.402 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.400+0000 7fe71bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:19.402 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-09-16T10:29:19.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.401+0000 7fe7389a0640 10 monclient: shutdown 2024-09-16T10:29:19.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.401+0000 7fe7389a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:19.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.401+0000 7fe7389a0640 1 -- 172.21.15.152:0/1918010697 >> v1:172.21.15.152:6791/0 conn(0x7fe7340a28d0 legacy=0x7fe73412b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.401+0000 7fe7389a0640 1 -- 172.21.15.152:0/1918010697 shutdown_connections 2024-09-16T10:29:19.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.401+0000 7fe7389a0640 1 -- 172.21.15.152:0/1918010697 >> 172.21.15.152:0/1918010697 conn(0x7fe7340a5cc0 msgr2=0x7fe734126490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:19.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.401+0000 7fe7389a0640 1 -- 172.21.15.152:0/1918010697 shutdown_connections 2024-09-16T10:29:19.403 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.401+0000 7fe7389a0640 1 -- 172.21.15.152:0/1918010697 wait complete. 2024-09-16T10:29:19.413 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 6f4cedf2-071d-44f9-87a3-a766668fcd03 2 2024-09-16T10:29:19.507 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:19.505+0000 7fb443923040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.509+0000 7f2d44e2b640 10 monclient: get_monmap_and_config 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.509+0000 7f2d44e2b640 10 monclient: build_initial_monmap 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.509+0000 7f2d44e2b640 10 monclient: monmap: 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:29:19.510243+0000 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:29:19.510243+0000 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:19.511 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:19.512 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:19.512 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:19.512 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:19.512 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:19.512 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:19.512 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.513 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 1 Processor -- start 2024-09-16T10:29:19.513 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 1 -- start start 2024-09-16T10:29:19.513 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 10 monclient: init 2024-09-16T10:29:19.513 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:19.513 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 10 monclient: _add_conns ranks=[1,4,3,5,8,7,2,6,0] 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 10 monclient(hunting): picked mon.noname-f con 0x7f2d400a2140 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 10 monclient(hunting): picked mon.noname-c con 0x7f2d400a15e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 10 monclient(hunting): picked mon.noname-g con 0x7f2d400a19f0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d40061ae0 con 0x7f2d400a2140 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d400a0920 con 0x7f2d400a19f0 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d400a0360 con 0x7f2d400a15e0 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d44e2b640 10 monclient(hunting): authenticate will time out at 715.167757s 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d3e59c640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f2d400a2140 0x7f2d400a2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:43418/0 (socket says 172.21.15.152:43418) 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d3e59c640 1 -- 172.21.15.152:0/2947439959 learned_addr learned my addr 172.21.15.152:0/2947439959 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:19.514 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.510+0000 7f2d3ed9d640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7f2d400a19f0 0x7f2d4013ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.152:44106/0 (socket says 172.21.15.152:44106) 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3ed9d640 1 -- 172.21.15.152:0/2947439959 >> v1:172.21.15.152:6790/0 conn(0x7f2d400a19f0 legacy=0x7f2d4013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3e59c640 1 -- 172.21.15.152:0/2947439959 >> v1:172.21.15.152:6789/0 conn(0x7f2d400a2140 legacy=0x7f2d400a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3dd9b640 1 -- 172.21.15.152:0/2947439959 >> v1:172.21.15.94:6791/0 conn(0x7f2d400a15e0 legacy=0x7f2d4013b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328206978 0 0) 0x7f2d400a0920 con 0x7f2d400a19f0 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): my global_id is 4125 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d28003520 con 0x7f2d400a19f0 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228143990 0 0) 0x7f2d40061ae0 con 0x7f2d400a2140 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.519 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): my global_id is 4105 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d400a0920 con 0x7f2d400a2140 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734820781 0 0) 0x7f2d400a0360 con 0x7f2d400a15e0 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient(hunting): my global_id is 4108 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d40061ae0 con 0x7f2d400a15e0 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3830967521 0 0) 0x7f2d28003520 con 0x7f2d400a19f0 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 >> v1:172.21.15.94:6791/0 conn(0x7f2d400a15e0 legacy=0x7f2d4013b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 >> v1:172.21.15.152:6789/0 conn(0x7f2d400a2140 legacy=0x7f2d400a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: _finish_hunting 0 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 monclient: found mon.noname-g 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.152:6790/0 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d400a0640 con 0x7f2d400a19f0 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: _finish_auth 0 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: _check_auth_tickets 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2d38001de0 con 0x7f2d400a19f0 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: dump: 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.521 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.522 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d44e2b640 5 monclient: authenticate success, global_id 4125 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d44e2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 <== mon.3 v1:172.21.15.152:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d380020f0 con 0x7f2d400a19f0 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 1 -- 172.21.15.152:0/2947439959 <== mon.3 v1:172.21.15.152:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2d38004ea0 con 0x7f2d400a19f0 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d44e2b640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d44e2b640 4 set_mon_vals no callback set 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.513+0000 7f2d3d59a640 10 monclient: dump: 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.523 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.524 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.514+0000 7f2d44e2b640 10 monclient: shutdown 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.514+0000 7f2d44e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.514+0000 7f2d44e2b640 1 -- 172.21.15.152:0/2947439959 >> v1:172.21.15.152:6790/0 conn(0x7f2d400a19f0 legacy=0x7f2d4013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.514+0000 7f2d44e2b640 1 -- 172.21.15.152:0/2947439959 shutdown_connections 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.514+0000 7f2d44e2b640 1 -- 172.21.15.152:0/2947439959 >> 172.21.15.152:0/2947439959 conn(0x7f2d400a93c0 msgr2=0x7f2d400a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.514+0000 7f2d44e2b640 1 -- 172.21.15.152:0/2947439959 shutdown_connections 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.514+0000 7f2d44e2b640 1 -- 172.21.15.152:0/2947439959 wait complete. 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.515+0000 7f2d44e2b640 10 monclient: build_initial_monmap 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.515+0000 7f2d44e2b640 10 monclient: monmap: 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-09-16T10:29:19.525 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.515+0000 7f2d44e2b640 1 Processor -- start 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.515+0000 7f2d44e2b640 1 -- start start 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.515+0000 7f2d44e2b640 10 monclient: init 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.515+0000 7f2d44e2b640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d44e2b640 10 monclient: _add_conns ranks=[7,8,1,2,3,4,6,5,0] 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d44e2b640 10 monclient(hunting): picked mon.noname-i con 0x7f2d400a15e0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d44e2b640 10 monclient(hunting): picked mon.noname-e con 0x7f2d400a19f0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d44e2b640 10 monclient(hunting): picked mon.noname-f con 0x7f2d400a2140 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d44e2b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d40120110 con 0x7f2d400a2140 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d44e2b640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d400a78f0 con 0x7f2d400a15e0 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d44e2b640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d400a73a0 con 0x7f2d400a19f0 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d44e2b640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d44e2b640 10 monclient(hunting): authenticate will time out at 715.173124s 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d3e59c640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7f2d400a15e0 0x7f2d4013e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.152:52548/0 (socket says 172.21.15.152:52548) 2024-09-16T10:29:19.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.516+0000 7f2d3e59c640 1 -- 172.21.15.152:0/3396187997 learned_addr learned my addr 172.21.15.152:0/3396187997 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d3e59c640 1 -- 172.21.15.152:0/3396187997 >> v1:172.21.15.152:6792/0 conn(0x7f2d400a15e0 legacy=0x7f2d4013e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d3ed9d640 1 -- 172.21.15.152:0/3396187997 >> v1:172.21.15.152:6789/0 conn(0x7f2d400a2140 legacy=0x7f2d401d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d3dd9b640 1 -- 172.21.15.152:0/3396187997 >> v1:172.21.15.94:6793/0 conn(0x7f2d400a19f0 legacy=0x7f2d4013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046572861 0 0) 0x7f2d400a78f0 con 0x7f2d400a15e0 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): my global_id is 4129 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d20003520 con 0x7f2d400a15e0 2024-09-16T10:29:19.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544171691 0 0) 0x7f2d40120110 con 0x7f2d400a2140 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): my global_id is 4114 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d400a78f0 con 0x7f2d400a2140 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364535938 0 0) 0x7f2d400a73a0 con 0x7f2d400a19f0 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): my global_id is 4148 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d40120110 con 0x7f2d400a19f0 2024-09-16T10:29:19.532 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2727578250 0 0) 0x7f2d20003520 con 0x7f2d400a15e0 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d400a73a0 con 0x7f2d400a15e0 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2d30001c90 con 0x7f2d400a15e0 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): dump: 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.533 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 323676594 0 0) 0x7f2d400a78f0 con 0x7f2d400a2140 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d20003520 con 0x7f2d400a2140 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2d38004ea0 con 0x7f2d400a2140 2024-09-16T10:29:19.534 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): dump: 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.535 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4267925920 0 0) 0x7f2d40120110 con 0x7f2d400a19f0 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d400a78f0 con 0x7f2d400a19f0 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2d34001d10 con 0x7f2d400a19f0 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 10 monclient(hunting): dump: 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.536 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.537 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2493937013 0 0) 0x7f2d400a73a0 con 0x7f2d400a15e0 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 >> v1:172.21.15.94:6793/0 conn(0x7f2d400a19f0 legacy=0x7f2d4013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.518+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 >> v1:172.21.15.152:6789/0 conn(0x7f2d400a2140 legacy=0x7f2d401d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 1 monclient: found mon.i 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d40121680 con 0x7f2d400a15e0 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 10 monclient: _finish_auth 0 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 5 monclient: authenticate success, global_id 4129 2024-09-16T10:29:19.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 10 monclient: _renew_subs 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4013f500 con 0x7f2d400a15e0 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 10 monclient: _renew_subs 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d401213a0 con 0x7f2d400a15e0 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d30004dd0 con 0x7f2d400a15e0 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2d30005280 con 0x7f2d400a15e0 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 10 monclient: dump: 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.539 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.540 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d3cd99640 4 set_mon_vals no callback set 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f2d300056e0 con 0x7f2d400a15e0 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1361201393 0 0) 0x7f2d300069a0 con 0x7f2d400a15e0 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.519+0000 7f2d44e2b640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d24005040 con 0x7f2d400a15e0 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.522+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f2d30006690 con 0x7f2d400a15e0 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.522+0000 7f2d2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.522+0000 7f2d2effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:19.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.589+0000 7f2d44e2b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6f4cedf2-071d-44f9-87a3-a766668fcd03", "id": 2}] 2024-09-16T10:29:19.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.589+0000 7f2d44e2b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6f4cedf2-071d-44f9-87a3-a766668fcd03", "id": 2}] 2024-09-16T10:29:19.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.589+0000 7f2d44e2b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.589+0000 7f2d44e2b640 1 -- 172.21.15.152:0/3396187997 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "osd new", "uuid": "6f4cedf2-071d-44f9-87a3-a766668fcd03", "id": 2} v 0) v1 -- 0x7f2d24005870 con 0x7f2d400a15e0 2024-09-16T10:29:19.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.593+0000 7f2d2effd640 1 -- 172.21.15.152:0/3396187997 <== mon.7 v1:172.21.15.152:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6f4cedf2-071d-44f9-87a3-a766668fcd03", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 757502798 0 216599125) 0x7f2d30005b20 con 0x7f2d400a15e0 2024-09-16T10:29:19.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.593+0000 7f2d2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6f4cedf2-071d-44f9-87a3-a766668fcd03", "id": 2}] 2024-09-16T10:29:19.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.593+0000 7f2d2effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:19.595 INFO:teuthology.orchestra.run.smithi152.stdout:2 2024-09-16T10:29:19.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.594+0000 7f2d44e2b640 10 monclient: shutdown 2024-09-16T10:29:19.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.594+0000 7f2d44e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:19.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.594+0000 7f2d44e2b640 1 -- 172.21.15.152:0/3396187997 >> v1:172.21.15.152:6792/0 conn(0x7f2d400a15e0 legacy=0x7f2d4013e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.594+0000 7f2d44e2b640 1 -- 172.21.15.152:0/3396187997 shutdown_connections 2024-09-16T10:29:19.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.594+0000 7f2d44e2b640 1 -- 172.21.15.152:0/3396187997 >> 172.21.15.152:0/3396187997 conn(0x7f2d400a93c0 msgr2=0x7f2d400a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:19.597 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.594+0000 7f2d44e2b640 1 -- 172.21.15.152:0/3396187997 shutdown_connections 2024-09-16T10:29:19.597 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.594+0000 7f2d44e2b640 1 -- 172.21.15.152:0/3396187997 wait complete. 2024-09-16T10:29:19.597 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:19.595+0000 7fb443923040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T10:29:19.606 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 3634242e-61e0-4221-8c38-7dce4d5053ec 3 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.700+0000 7ff7583bc640 10 monclient: get_monmap_and_config 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.700+0000 7ff7583bc640 10 monclient: build_initial_monmap 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.700+0000 7ff7583bc640 10 monclient: monmap: 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:29:19.701156+0000 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:29:19.701156+0000 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.702 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:19.703 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:19.703 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:19.703 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:19.703 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:19.703 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:19.703 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:19.703 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:19.703 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:19.703 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.700+0000 7ff7583bc640 1 Processor -- start 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 1 -- start start 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 10 monclient: init 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 10 monclient: _add_conns ranks=[3,4,7,8,5,0,1,6,2] 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 10 monclient(hunting): picked mon.noname-g con 0x7ff7500a2140 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 10 monclient(hunting): picked mon.noname-c con 0x7ff7500a15e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 10 monclient(hunting): picked mon.noname-i con 0x7ff7500a19f0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff750061ae0 con 0x7ff7500a2140 2024-09-16T10:29:19.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7500a0920 con 0x7ff7500a15e0 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7500a0360 con 0x7ff7500a19f0 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff7583bc640 10 monclient(hunting): authenticate will time out at 715.358699s 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff756158640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7ff7500a2140 0x7ff7500a2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.152:44112/0 (socket says 172.21.15.152:44112) 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.701+0000 7ff756158640 1 -- 172.21.15.152:0/577898661 learned_addr learned my addr 172.21.15.152:0/577898661 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.703+0000 7ff756158640 1 -- 172.21.15.152:0/577898661 >> v1:172.21.15.152:6790/0 conn(0x7ff7500a2140 legacy=0x7ff7500a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.703+0000 7ff756959640 1 -- 172.21.15.152:0/577898661 >> v1:172.21.15.152:6792/0 conn(0x7ff7500a19f0 legacy=0x7ff75013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304895783 0 0) 0x7ff750061ae0 con 0x7ff7500a2140 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient(hunting): my global_id is 4134 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff73c003520 con 0x7ff7500a2140 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210295319 0 0) 0x7ff7500a0360 con 0x7ff7500a19f0 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient(hunting): my global_id is 4138 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755957640 1 -- 172.21.15.152:0/577898661 >> v1:172.21.15.94:6791/0 conn(0x7ff7500a15e0 legacy=0x7ff75013b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff750061ae0 con 0x7ff7500a19f0 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2139567893 0 0) 0x7ff73c003520 con 0x7ff7500a2140 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 >> v1:172.21.15.152:6792/0 conn(0x7ff7500a19f0 legacy=0x7ff75013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 >> v1:172.21.15.94:6791/0 conn(0x7ff7500a15e0 legacy=0x7ff75013b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: _finish_hunting 0 2024-09-16T10:29:19.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 monclient: found mon.noname-g 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.152:6790/0 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7500a0640 con 0x7ff7500a2140 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: _finish_auth 0 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: _check_auth_tickets 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff7583bc640 5 monclient: authenticate success, global_id 4134 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff7583bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff74c001a60 con 0x7ff7500a2140 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:29:19.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: dump: 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.711 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff7583bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 <== mon.3 v1:172.21.15.152:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff74c001d70 con 0x7ff7500a2140 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff7583bc640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 1 -- 172.21.15.152:0/577898661 <== mon.3 v1:172.21.15.152:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff74c004be0 con 0x7ff7500a2140 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff7583bc640 4 set_mon_vals no callback set 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.704+0000 7ff755156640 10 monclient: dump: 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.712 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.713 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.705+0000 7ff7583bc640 10 monclient: shutdown 2024-09-16T10:29:19.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.705+0000 7ff7583bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:19.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.705+0000 7ff7583bc640 1 -- 172.21.15.152:0/577898661 >> v1:172.21.15.152:6790/0 conn(0x7ff7500a2140 legacy=0x7ff7500a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.705+0000 7ff7583bc640 1 -- 172.21.15.152:0/577898661 shutdown_connections 2024-09-16T10:29:19.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.705+0000 7ff7583bc640 1 -- 172.21.15.152:0/577898661 >> 172.21.15.152:0/577898661 conn(0x7ff7500a93c0 msgr2=0x7ff7500a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:19.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.705+0000 7ff7583bc640 1 -- 172.21.15.152:0/577898661 shutdown_connections 2024-09-16T10:29:19.714 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.705+0000 7ff7583bc640 1 -- 172.21.15.152:0/577898661 wait complete. 2024-09-16T10:29:19.714 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:19.705+0000 7fb443923040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T10:29:19.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.705+0000 7ff7583bc640 10 monclient: build_initial_monmap 2024-09-16T10:29:19.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.705+0000 7ff7583bc640 10 monclient: monmap: 2024-09-16T10:29:19.719 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:19.719 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:19.719 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-09-16T10:29:19.719 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-09-16T10:29:19.719 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.720 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 1 Processor -- start 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 1 -- start start 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 10 monclient: init 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 10 monclient: _add_conns ranks=[4,7,1,3,8,2,5,0,6] 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 10 monclient(hunting): picked mon.noname-c con 0x7ff7500a15e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 10 monclient(hunting): picked mon.noname-i con 0x7ff7500a19f0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 10 monclient(hunting): picked mon.noname-f con 0x7ff7500a2140 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75013f2e0 con 0x7ff7500a2140 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7500a77a0 con 0x7ff7500a15e0 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7501203c0 con 0x7ff7500a19f0 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff7583bc640 10 monclient(hunting): authenticate will time out at 715.363712s 2024-09-16T10:29:19.721 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff755957640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7ff7500a19f0 0x7ff75013dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.152:52570/0 (socket says 172.21.15.152:52570) 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.706+0000 7ff755957640 1 -- 172.21.15.152:0/922929715 learned_addr learned my addr 172.21.15.152:0/922929715 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.708+0000 7ff756959640 1 -- 172.21.15.152:0/922929715 >> v1:172.21.15.152:6789/0 conn(0x7ff7500a2140 legacy=0x7ff7501d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.708+0000 7ff755957640 1 -- 172.21.15.152:0/922929715 >> v1:172.21.15.152:6792/0 conn(0x7ff7500a19f0 legacy=0x7ff75013dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff756158640 1 -- 172.21.15.152:0/922929715 >> v1:172.21.15.94:6791/0 conn(0x7ff7500a15e0 legacy=0x7ff75013d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483838161 0 0) 0x7ff7501203c0 con 0x7ff7500a19f0 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): my global_id is 4147 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff734003520 con 0x7ff7500a19f0 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129687097 0 0) 0x7ff75013f2e0 con 0x7ff7500a2140 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.722 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): my global_id is 4123 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7501203c0 con 0x7ff7500a2140 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477742628 0 0) 0x7ff7500a77a0 con 0x7ff7500a15e0 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): my global_id is 4126 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff75013f2e0 con 0x7ff7500a15e0 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1855058631 0 0) 0x7ff734003520 con 0x7ff7500a19f0 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7500a77a0 con 0x7ff7500a19f0 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff740003610 con 0x7ff7500a19f0 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.723 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): dump: 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.724 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2752114675 0 0) 0x7ff7501203c0 con 0x7ff7500a2140 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff734003520 con 0x7ff7500a2140 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff748003cd0 con 0x7ff7500a2140 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): dump: 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.725 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.726 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3193474665 0 0) 0x7ff75013f2e0 con 0x7ff7500a15e0 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7501203c0 con 0x7ff7500a15e0 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff74c001320 con 0x7ff7500a15e0 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient(hunting): dump: 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.727 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1765698570 0 0) 0x7ff7500a77a0 con 0x7ff7500a19f0 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 >> v1:172.21.15.94:6791/0 conn(0x7ff7500a15e0 legacy=0x7ff75013d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 >> v1:172.21.15.152:6789/0 conn(0x7ff7500a2140 legacy=0x7ff7501d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 monclient: found mon.i 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7501206a0 con 0x7ff7500a19f0 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff7583bc640 5 monclient: authenticate success, global_id 4147 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff7583bc640 10 monclient: _renew_subs 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff7583bc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff7583bc640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75009f6a0 con 0x7ff7500a19f0 2024-09-16T10:29:19.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff7583bc640 10 monclient: _renew_subs 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff7583bc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff7583bc640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75013e480 con 0x7ff7500a19f0 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff740003b70 con 0x7ff7500a19f0 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff740004020 con 0x7ff7500a19f0 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff754955640 4 set_mon_vals no callback set 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 10 monclient: dump: 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.730 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.731 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7ff7400057c0 con 0x7ff7500a19f0 2024-09-16T10:29:19.732 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.709+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3243250976 0 0) 0x7ff7400069f0 con 0x7ff7500a19f0 2024-09-16T10:29:19.732 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.710+0000 7ff7583bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.732 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.710+0000 7ff7583bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.732 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.710+0000 7ff7583bc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.732 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.710+0000 7ff7583bc640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff738005040 con 0x7ff7500a19f0 2024-09-16T10:29:19.732 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.712+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7ff740006670 con 0x7ff7500a19f0 2024-09-16T10:29:19.732 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.712+0000 7ff746ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.732 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.712+0000 7ff746ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:19.754 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:19.752+0000 7fb443923040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T10:29:19.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.779+0000 7ff7583bc640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3634242e-61e0-4221-8c38-7dce4d5053ec", "id": 3}] 2024-09-16T10:29:19.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.779+0000 7ff7583bc640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3634242e-61e0-4221-8c38-7dce4d5053ec", "id": 3}] 2024-09-16T10:29:19.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.779+0000 7ff7583bc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:19.781 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.779+0000 7ff7583bc640 1 -- 172.21.15.152:0/922929715 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "osd new", "uuid": "3634242e-61e0-4221-8c38-7dce4d5053ec", "id": 3} v 0) v1 -- 0x7ff7380052f0 con 0x7ff7500a19f0 2024-09-16T10:29:19.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.783+0000 7ff746ffd640 1 -- 172.21.15.152:0/922929715 <== mon.7 v1:172.21.15.152:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3634242e-61e0-4221-8c38-7dce4d5053ec", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3876287401 0 525046306) 0x7ff740005b10 con 0x7ff7500a19f0 2024-09-16T10:29:19.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.783+0000 7ff746ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3634242e-61e0-4221-8c38-7dce4d5053ec", "id": 3}] 2024-09-16T10:29:19.785 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.783+0000 7ff746ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:19.785 INFO:teuthology.orchestra.run.smithi152.stdout:3 2024-09-16T10:29:19.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.784+0000 7ff7583bc640 10 monclient: shutdown 2024-09-16T10:29:19.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.784+0000 7ff7583bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:19.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.784+0000 7ff7583bc640 1 -- 172.21.15.152:0/922929715 >> v1:172.21.15.152:6792/0 conn(0x7ff7500a19f0 legacy=0x7ff75013dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.784+0000 7ff7583bc640 1 -- 172.21.15.152:0/922929715 shutdown_connections 2024-09-16T10:29:19.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.784+0000 7ff7583bc640 1 -- 172.21.15.152:0/922929715 >> 172.21.15.152:0/922929715 conn(0x7ff7500a93c0 msgr2=0x7ff7500a7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:19.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.784+0000 7ff7583bc640 1 -- 172.21.15.152:0/922929715 shutdown_connections 2024-09-16T10:29:19.786 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.784+0000 7ff7583bc640 1 -- 172.21.15.152:0/922929715 wait complete. 2024-09-16T10:29:19.796 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new eed750dc-2d29-443f-a91c-d9c5abdcf3d2 4 2024-09-16T10:29:19.870 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:19.868+0000 7fb443923040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.891+0000 7f60017b5640 10 monclient: get_monmap_and_config 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.891+0000 7f60017b5640 10 monclient: build_initial_monmap 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.891+0000 7f60017b5640 10 monclient: monmap: 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:29:19.891992+0000 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:29:19.891992+0000 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.893 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.891+0000 7f60017b5640 1 Processor -- start 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.891+0000 7f60017b5640 1 -- start start 2024-09-16T10:29:19.894 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.891+0000 7f60017b5640 10 monclient: init 2024-09-16T10:29:19.899 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:19.899 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 10 monclient: _add_conns ranks=[2,8,0,6,3,5,7,4,1] 2024-09-16T10:29:19.899 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 10 monclient(hunting): picked mon.noname-b con 0x7f5ffc0a2140 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 10 monclient(hunting): picked mon.noname-e con 0x7f5ffc0a15e0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 10 monclient(hunting): picked mon.noname-a con 0x7f5ffc0a19f0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc061ae0 con 0x7f5ffc0a19f0 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc0a0920 con 0x7f5ffc0a2140 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc0a0360 con 0x7f5ffc0a15e0 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f60017b5640 10 monclient(hunting): authenticate will time out at 715.549373s 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f5ffa7fc640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f5ffc0a15e0 0x7f5ffc13b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.152:52182/0 (socket says 172.21.15.152:52182) 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f5ffa7fc640 1 -- 172.21.15.152:0/3143784816 learned_addr learned my addr 172.21.15.152:0/3143784816 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.892+0000 7f5ffb7fe640 1 --1- 172.21.15.152:0/3143784816 >> v1:172.21.15.94:6789/0 conn(0x7f5ffc0a19f0 0x7f5ffc13ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.152:33052/0 (socket says 172.21.15.152:33052) 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.894+0000 7f5ffb7fe640 1 -- 172.21.15.152:0/3143784816 >> v1:172.21.15.94:6789/0 conn(0x7f5ffc0a19f0 legacy=0x7f5ffc13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.894+0000 7f5ffaffd640 1 -- 172.21.15.152:0/3143784816 >> v1:172.21.15.94:6790/0 conn(0x7f5ffc0a2140 legacy=0x7f5ffc0a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.894+0000 7f5ffa7fc640 1 -- 172.21.15.152:0/3143784816 >> v1:172.21.15.94:6793/0 conn(0x7f5ffc0a15e0 legacy=0x7f5ffc13b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355283422 0 0) 0x7f5ffc061ae0 con 0x7f5ffc0a19f0 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): my global_id is 4131 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe4003520 con 0x7f5ffc0a19f0 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652612031 0 0) 0x7f5ffc0a0920 con 0x7f5ffc0a2140 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): my global_id is 4133 2024-09-16T10:29:19.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc061ae0 con 0x7f5ffc0a2140 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775032889 0 0) 0x7f5ffc0a0360 con 0x7f5ffc0a15e0 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient(hunting): my global_id is 4157 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc0a0920 con 0x7f5ffc0a15e0 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3700642640 0 0) 0x7f5ffc061ae0 con 0x7f5ffc0a2140 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 >> v1:172.21.15.94:6793/0 conn(0x7f5ffc0a15e0 legacy=0x7f5ffc13b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 >> v1:172.21.15.94:6789/0 conn(0x7f5ffc0a19f0 legacy=0x7f5ffc13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 monclient: found mon.noname-b 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:19.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc0a0640 con 0x7f5ffc0a2140 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: _finish_auth 0 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5ff0001b70 con 0x7f5ffc0a2140 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: dump: 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.903 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f60017b5640 5 monclient: authenticate success, global_id 4133 2024-09-16T10:29:19.904 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f60017b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff0003610 con 0x7f5ffc0a2140 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f60017b5640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 1 -- 172.21.15.152:0/3143784816 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5ff0003b80 con 0x7f5ffc0a2140 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f60017b5640 4 set_mon_vals no callback set 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.895+0000 7f5ff9ffb640 10 monclient: dump: 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.905 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.906 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.896+0000 7f60017b5640 10 monclient: shutdown 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.896+0000 7f60017b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.896+0000 7f60017b5640 1 -- 172.21.15.152:0/3143784816 >> v1:172.21.15.94:6790/0 conn(0x7f5ffc0a2140 legacy=0x7f5ffc0a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.896+0000 7f60017b5640 1 -- 172.21.15.152:0/3143784816 shutdown_connections 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.896+0000 7f60017b5640 1 -- 172.21.15.152:0/3143784816 >> 172.21.15.152:0/3143784816 conn(0x7f5ffc0a93c0 msgr2=0x7f5ffc0a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.896+0000 7f60017b5640 1 -- 172.21.15.152:0/3143784816 shutdown_connections 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.896+0000 7f60017b5640 1 -- 172.21.15.152:0/3143784816 wait complete. 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.896+0000 7f60017b5640 10 monclient: build_initial_monmap 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.896+0000 7f60017b5640 10 monclient: monmap: 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-09-16T10:29:19.907 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-09-16T10:29:19.908 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:19.908 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.908 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:19.908 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:19.908 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:19.908 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:19.908 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:19.908 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:19.908 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 1 Processor -- start 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 1 -- start start 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 10 monclient: init 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 10 monclient: _add_conns ranks=[2,7,3,4,8,0,1,6,5] 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 10 monclient(hunting): picked mon.noname-b con 0x7f5ffc0a15e0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 10 monclient(hunting): picked mon.noname-i con 0x7f5ffc0a19f0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 10 monclient(hunting): picked mon.noname-g con 0x7f5ffc0a2140 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc09f520 con 0x7f5ffc0a15e0 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc121850 con 0x7f5ffc0a2140 2024-09-16T10:29:19.909 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc09efb0 con 0x7f5ffc0a19f0 2024-09-16T10:29:19.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:19.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f60017b5640 10 monclient(hunting): authenticate will time out at 715.554704s 2024-09-16T10:29:19.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f5ffa7fc640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7f5ffc0a19f0 0x7f5ffc13f240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.152:52582/0 (socket says 172.21.15.152:52582) 2024-09-16T10:29:19.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f5ffb7fe640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7f5ffc0a2140 0x7f5ffc1d31c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.152:44122/0 (socket says 172.21.15.152:44122) 2024-09-16T10:29:19.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.897+0000 7f5ffa7fc640 1 -- 172.21.15.152:0/3208071516 learned_addr learned my addr 172.21.15.152:0/3208071516 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.899+0000 7f5ffa7fc640 1 -- 172.21.15.152:0/3208071516 >> v1:172.21.15.152:6792/0 conn(0x7f5ffc0a19f0 legacy=0x7f5ffc13f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.899+0000 7f5ffb7fe640 1 -- 172.21.15.152:0/3208071516 >> v1:172.21.15.152:6790/0 conn(0x7f5ffc0a2140 legacy=0x7f5ffc1d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5ffaffd640 1 -- 172.21.15.152:0/3208071516 >> v1:172.21.15.94:6790/0 conn(0x7f5ffc0a15e0 legacy=0x7f5ffc13eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349502420 0 0) 0x7f5ffc121850 con 0x7f5ffc0a2140 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): my global_id is 4143 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe8003520 con 0x7f5ffc0a2140 2024-09-16T10:29:19.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397281064 0 0) 0x7f5ffc09efb0 con 0x7f5ffc0a19f0 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): my global_id is 4156 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc121850 con 0x7f5ffc0a19f0 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516565264 0 0) 0x7f5ffc09f520 con 0x7f5ffc0a15e0 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): my global_id is 4142 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc09efb0 con 0x7f5ffc0a15e0 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1151146642 0 0) 0x7f5fe8003520 con 0x7f5ffc0a2140 2024-09-16T10:29:19.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ffc09f520 con 0x7f5ffc0a2140 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5ff4001ca0 con 0x7f5ffc0a2140 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): dump: 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.916 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477360611 0 0) 0x7f5ffc121850 con 0x7f5ffc0a19f0 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe8003520 con 0x7f5ffc0a19f0 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5fec001cb0 con 0x7f5ffc0a19f0 2024-09-16T10:29:19.917 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): dump: 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.918 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3520058435 0 0) 0x7f5ffc09efb0 con 0x7f5ffc0a15e0 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ffc121850 con 0x7f5ffc0a15e0 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5ff0003ac0 con 0x7f5ffc0a15e0 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient(hunting): dump: 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.919 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.920 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3559935023 0 0) 0x7f5ffc09f520 con 0x7f5ffc0a2140 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 >> v1:172.21.15.152:6792/0 conn(0x7f5ffc0a19f0 legacy=0x7f5ffc13f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 >> v1:172.21.15.94:6790/0 conn(0x7f5ffc0a15e0 legacy=0x7f5ffc13eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 monclient: found mon.g 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc0a7620 con 0x7f5ffc0a2140 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:19.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f60017b5640 5 monclient: authenticate success, global_id 4143 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f60017b5640 10 monclient: _renew_subs 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f60017b5640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f60017b5640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ffc121570 con 0x7f5ffc0a2140 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f60017b5640 10 monclient: _renew_subs 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f60017b5640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f60017b5640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ffc11f4f0 con 0x7f5ffc0a2140 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff4001e00 con 0x7f5ffc0a2140 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5ff4005490 con 0x7f5ffc0a2140 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5ff97fa640 4 set_mon_vals no callback set 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 10 monclient: dump: 2024-09-16T10:29:19.922 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:19.923 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f5ff40058f0 con 0x7f5ffc0a2140 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.900+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2027532580 0 0) 0x7f5ff4006da0 con 0x7f5ffc0a2140 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.901+0000 7f60017b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.901+0000 7f60017b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.901+0000 7f60017b5640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.901+0000 7f60017b5640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.152:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc005040 con 0x7f5ffc0a2140 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.903+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f5ff4006900 con 0x7f5ffc0a2140 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.903+0000 7f5fe37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:19.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.903+0000 7f5fe37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:19.925 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:19.923+0000 7fb443923040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T10:29:19.969 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:19.968+0000 7fb443923040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T10:29:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.969+0000 7f60017b5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eed750dc-2d29-443f-a91c-d9c5abdcf3d2", "id": 4}] 2024-09-16T10:29:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.969+0000 7f60017b5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eed750dc-2d29-443f-a91c-d9c5abdcf3d2", "id": 4}] 2024-09-16T10:29:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.969+0000 7f60017b5640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.969+0000 7f60017b5640 1 -- 172.21.15.152:0/3208071516 --> v1:172.21.15.152:6790/0 -- mon_command({"prefix": "osd new", "uuid": "eed750dc-2d29-443f-a91c-d9c5abdcf3d2", "id": 4} v 0) v1 -- 0x7f5fdc005870 con 0x7f5ffc0a2140 2024-09-16T10:29:19.994 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.992+0000 7f5fe37fe640 1 -- 172.21.15.152:0/3208071516 <== mon.3 v1:172.21.15.152:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eed750dc-2d29-443f-a91c-d9c5abdcf3d2", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3702057276 0 1697012583) 0x7f5ff4005d30 con 0x7f5ffc0a2140 2024-09-16T10:29:19.994 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.992+0000 7f5fe37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eed750dc-2d29-443f-a91c-d9c5abdcf3d2", "id": 4}] 2024-09-16T10:29:19.994 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.992+0000 7f5fe37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:19.994 INFO:teuthology.orchestra.run.smithi152.stdout:4 2024-09-16T10:29:19.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.993+0000 7f60017b5640 10 monclient: shutdown 2024-09-16T10:29:19.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.993+0000 7f60017b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:19.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.993+0000 7f60017b5640 1 -- 172.21.15.152:0/3208071516 >> v1:172.21.15.152:6790/0 conn(0x7f5ffc0a2140 legacy=0x7f5ffc1d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:19.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.993+0000 7f60017b5640 1 -- 172.21.15.152:0/3208071516 shutdown_connections 2024-09-16T10:29:19.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.993+0000 7f60017b5640 1 -- 172.21.15.152:0/3208071516 >> 172.21.15.152:0/3208071516 conn(0x7f5ffc0a93c0 msgr2=0x7f5ffc0a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:19.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.993+0000 7f60017b5640 1 -- 172.21.15.152:0/3208071516 shutdown_connections 2024-09-16T10:29:19.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:19.993+0000 7f60017b5640 1 -- 172.21.15.152:0/3208071516 wait complete. 2024-09-16T10:29:20.005 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 5d6daac7-1ac7-4f0a-b315-27d1333fc3ea 5 2024-09-16T10:29:20.087 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:20.086+0000 7fb443923040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T10:29:20.102 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.100+0000 7f980e383640 10 monclient: get_monmap_and_config 2024-09-16T10:29:20.102 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.100+0000 7f980e383640 10 monclient: build_initial_monmap 2024-09-16T10:29:20.102 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.100+0000 7f980e383640 10 monclient: monmap: 2024-09-16T10:29:20.102 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:20.102 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.102 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:29:20.101478+0000 2024-09-16T10:29:20.102 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:29:20.101478+0000 2024-09-16T10:29:20.102 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:20.103 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.101+0000 7f980e383640 1 Processor -- start 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.101+0000 7f980e383640 1 -- start start 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.101+0000 7f980e383640 10 monclient: init 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.101+0000 7f980e383640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980e383640 10 monclient: _add_conns ranks=[8,3,1,6,0,4,7,5,2] 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980e383640 10 monclient(hunting): picked mon.noname-e con 0x7f98080a2140 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980e383640 10 monclient(hunting): picked mon.noname-g con 0x7f98080a15e0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980e383640 10 monclient(hunting): picked mon.noname-f con 0x7f98080a19f0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980e383640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9808061ae0 con 0x7f98080a19f0 2024-09-16T10:29:20.105 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980e383640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98080a0920 con 0x7f98080a15e0 2024-09-16T10:29:20.106 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980e383640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98080a0360 con 0x7f98080a2140 2024-09-16T10:29:20.106 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980e383640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:20.106 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980e383640 10 monclient(hunting): authenticate will time out at 715.759138s 2024-09-16T10:29:20.106 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f98077fe640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7f98080a15e0 0x7f980813b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.152:44132/0 (socket says 172.21.15.152:44132) 2024-09-16T10:29:20.106 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f98077fe640 1 -- 172.21.15.152:0/1559562356 learned_addr learned my addr 172.21.15.152:0/1559562356 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:20.106 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.102+0000 7f980c920640 1 --1- 172.21.15.152:0/1559562356 >> v1:172.21.15.152:6789/0 conn(0x7f98080a19f0 0x7f980813ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:43442/0 (socket says 172.21.15.152:43442) 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f98077fe640 1 -- 172.21.15.152:0/1559562356 >> v1:172.21.15.152:6790/0 conn(0x7f98080a15e0 legacy=0x7f980813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f980c920640 1 -- 172.21.15.152:0/1559562356 >> v1:172.21.15.152:6789/0 conn(0x7f98080a19f0 legacy=0x7f980813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395616086 0 0) 0x7f9808061ae0 con 0x7f98080a19f0 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): my global_id is 4132 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f4003520 con 0x7f98080a19f0 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9807fff640 1 -- 172.21.15.152:0/1559562356 >> v1:172.21.15.94:6793/0 conn(0x7f98080a2140 legacy=0x7f98080a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934539178 0 0) 0x7f98080a0920 con 0x7f98080a15e0 2024-09-16T10:29:20.110 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): my global_id is 4152 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9808061ae0 con 0x7f98080a15e0 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694810986 0 0) 0x7f98080a0360 con 0x7f98080a2140 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient(hunting): my global_id is 4166 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98080a0920 con 0x7f98080a2140 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3161119502 0 0) 0x7f97f4003520 con 0x7f98080a19f0 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 >> v1:172.21.15.94:6793/0 conn(0x7f98080a2140 legacy=0x7f98080a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.111 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 >> v1:172.21.15.152:6790/0 conn(0x7f98080a15e0 legacy=0x7f980813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 monclient: found mon.noname-f 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.152:6789/0 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98080a0640 con 0x7f98080a19f0 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9800001d10 con 0x7f98080a19f0 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f980e383640 5 monclient: authenticate success, global_id 4132 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f980e383640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:20.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f9806ffd640 10 monclient: dump: 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.113 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.104+0000 7f980e383640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9800002020 con 0x7f98080a19f0 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f9806ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f9806ffd640 1 -- 172.21.15.152:0/1559562356 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9800004ea0 con 0x7f98080a19f0 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f980e383640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f980e383640 4 set_mon_vals no callback set 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f9806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f9806ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f9806ffd640 10 monclient: dump: 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.115 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f980e383640 10 monclient: shutdown 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f980e383640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f980e383640 1 -- 172.21.15.152:0/1559562356 >> v1:172.21.15.152:6789/0 conn(0x7f98080a19f0 legacy=0x7f980813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f980e383640 1 -- 172.21.15.152:0/1559562356 shutdown_connections 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f980e383640 1 -- 172.21.15.152:0/1559562356 >> 172.21.15.152:0/1559562356 conn(0x7f98080a93c0 msgr2=0x7f98080a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f980e383640 1 -- 172.21.15.152:0/1559562356 shutdown_connections 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.105+0000 7f980e383640 1 -- 172.21.15.152:0/1559562356 wait complete. 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.106+0000 7f980e383640 10 monclient: build_initial_monmap 2024-09-16T10:29:20.116 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.106+0000 7f980e383640 10 monclient: monmap: 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 0 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 0.000000 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:created 0.000000 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:20.117 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.106+0000 7f980e383640 1 Processor -- start 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.106+0000 7f980e383640 1 -- start start 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.106+0000 7f980e383640 10 monclient: init 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 10 monclient: _add_conns ranks=[0,7,1,6,2,8,4,5,3] 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 10 monclient(hunting): picked mon.noname-a con 0x7f98080a15e0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 10 monclient(hunting): picked mon.noname-i con 0x7f98080a19f0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 10 monclient(hunting): picked mon.noname-f con 0x7f98080a2140 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9808121730 con 0x7f98080a15e0 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98080a73d0 con 0x7f98080a2140 2024-09-16T10:29:20.118 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980809f760 con 0x7f98080a19f0 2024-09-16T10:29:20.119 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:20.119 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980e383640 10 monclient(hunting): authenticate will time out at 715.764311s 2024-09-16T10:29:20.119 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980c920640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f98080a2140 0x7f980813f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.152:43452/0 (socket says 172.21.15.152:43452) 2024-09-16T10:29:20.119 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f98077fe640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7f98080a19f0 0x7f980813ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.152:52590/0 (socket says 172.21.15.152:52590) 2024-09-16T10:29:20.119 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.107+0000 7f980c920640 1 -- 172.21.15.152:0/1574148016 learned_addr learned my addr 172.21.15.152:0/1574148016 (peer_addr_for_me v1:172.21.15.152:0/0) 2024-09-16T10:29:20.122 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f98077fe640 1 -- 172.21.15.152:0/1574148016 >> v1:172.21.15.152:6792/0 conn(0x7f98080a19f0 legacy=0x7f980813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.122 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f980c920640 1 -- 172.21.15.152:0/1574148016 >> v1:172.21.15.152:6789/0 conn(0x7f98080a2140 legacy=0x7f980813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.122 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319109594 0 0) 0x7f980809f760 con 0x7f98080a19f0 2024-09-16T10:29:20.122 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 10 monclient(hunting): my global_id is 4165 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97e8003520 con 0x7f98080a19f0 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115222146 0 0) 0x7f98080a73d0 con 0x7f98080a2140 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 10 monclient(hunting): my global_id is 4141 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980809f760 con 0x7f98080a2140 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9807fff640 1 -- 172.21.15.152:0/1574148016 >> v1:172.21.15.94:6789/0 conn(0x7f98080a15e0 legacy=0x7f980813e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.123 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3609322251 0 0) 0x7f97e8003520 con 0x7f98080a19f0 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.109+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98080a73d0 con 0x7f98080a19f0 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f97f0001d60 con 0x7f98080a19f0 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): dump: 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.124 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2295918802 0 0) 0x7f980809f760 con 0x7f98080a2140 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97e8003520 con 0x7f98080a2140 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9800004ea0 con 0x7f98080a2140 2024-09-16T10:29:20.125 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): dump: 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.126 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829604824 0 0) 0x7f9808121730 con 0x7f98080a15e0 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient(hunting): my global_id is 4140 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980809f760 con 0x7f98080a15e0 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 877283279 0 0) 0x7f98080a73d0 con 0x7f98080a19f0 2024-09-16T10:29:20.127 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 >> v1:172.21.15.152:6789/0 conn(0x7f98080a2140 legacy=0x7f980813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 >> v1:172.21.15.94:6789/0 conn(0x7f98080a15e0 legacy=0x7f980813e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient: _finish_hunting 0 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 monclient: found mon.i 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98080a7920 con 0x7f98080a19f0 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient: _finish_auth 0 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f980e383640 5 monclient: authenticate success, global_id 4165 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f980e383640 10 monclient: _renew_subs 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f980e383640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:20.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f980e383640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980811f410 con 0x7f98080a19f0 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f980e383640 10 monclient: _renew_subs 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f980e383640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f980e383640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98081d41d0 con 0x7f98080a19f0 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97f0004c80 con 0x7f98080a19f0 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f97f0005130 con 0x7f98080a19f0 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f98067fc640 4 set_mon_vals no callback set 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 10 monclient: dump: 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:epoch 1 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.129 INFO:teuthology.orchestra.run.smithi152.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:election_strategy: 1 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.130 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T10:29:20.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f97f00055d0 con 0x7f98080a19f0 2024-09-16T10:29:20.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.110+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1840571368 0 0) 0x7f97f0006500 con 0x7f98080a19f0 2024-09-16T10:29:20.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.111+0000 7f980e383640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:20.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.111+0000 7f980e383640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:20.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.111+0000 7f980e383640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:20.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.111+0000 7f980e383640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec005040 con 0x7f98080a19f0 2024-09-16T10:29:20.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.113+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f97f0006710 con 0x7f98080a19f0 2024-09-16T10:29:20.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.113+0000 7f9804ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:20.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.113+0000 7f9804ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:20.140 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:20.138+0000 7fb443923040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T10:29:20.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.179+0000 7f980e383640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5d6daac7-1ac7-4f0a-b315-27d1333fc3ea", "id": 5}] 2024-09-16T10:29:20.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.179+0000 7f980e383640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5d6daac7-1ac7-4f0a-b315-27d1333fc3ea", "id": 5}] 2024-09-16T10:29:20.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.179+0000 7f980e383640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:20.181 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.179+0000 7f980e383640 1 -- 172.21.15.152:0/1574148016 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "osd new", "uuid": "5d6daac7-1ac7-4f0a-b315-27d1333fc3ea", "id": 5} v 0) v1 -- 0x7f97ec005870 con 0x7f98080a19f0 2024-09-16T10:29:20.191 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:20.190+0000 7fb443923040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T10:29:20.257 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:20.256+0000 7fb443923040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T10:29:20.300 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:20.299+0000 7fb443923040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T10:29:20.343 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:20.341+0000 7fb443923040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T10:29:20.771 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.770+0000 7f9804ff9640 1 -- 172.21.15.152:0/1574148016 <== mon.7 v1:172.21.15.152:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5d6daac7-1ac7-4f0a-b315-27d1333fc3ea", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2577207166 0 1988412176) 0x7f97f00068b0 con 0x7f98080a19f0 2024-09-16T10:29:20.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.770+0000 7f9804ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5d6daac7-1ac7-4f0a-b315-27d1333fc3ea", "id": 5}] 2024-09-16T10:29:20.772 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.770+0000 7f9804ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:20.772 INFO:teuthology.orchestra.run.smithi152.stdout:5 2024-09-16T10:29:20.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.771+0000 7f980e383640 10 monclient: shutdown 2024-09-16T10:29:20.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.771+0000 7f980e383640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:20.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.771+0000 7f980e383640 1 -- 172.21.15.152:0/1574148016 >> v1:172.21.15.152:6792/0 conn(0x7f98080a19f0 legacy=0x7f980813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.771+0000 7f980e383640 1 -- 172.21.15.152:0/1574148016 shutdown_connections 2024-09-16T10:29:20.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.771+0000 7f980e383640 1 -- 172.21.15.152:0/1574148016 >> 172.21.15.152:0/1574148016 conn(0x7f98080a93c0 msgr2=0x7f98080a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:20.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.771+0000 7f980e383640 1 -- 172.21.15.152:0/1574148016 shutdown_connections 2024-09-16T10:29:20.773 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T10:29:20.771+0000 7f980e383640 1 -- 172.21.15.152:0/1574148016 wait complete. 2024-09-16T10:29:20.783 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T10:29:20.783 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T10:29:20.786 INFO:tasks.ceph.osd.0:Started 2024-09-16T10:29:20.786 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T10:29:20.786 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T10:29:20.788 INFO:tasks.ceph.osd.1:Started 2024-09-16T10:29:20.788 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T10:29:20.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T10:29:20.791 INFO:tasks.ceph.osd.2:Started 2024-09-16T10:29:20.791 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T10:29:20.791 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T10:29:20.793 INFO:tasks.ceph.osd.3:Started 2024-09-16T10:29:20.793 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T10:29:20.793 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T10:29:20.795 INFO:tasks.ceph.osd.4:Started 2024-09-16T10:29:20.796 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T10:29:20.796 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T10:29:20.798 INFO:tasks.ceph.osd.5:Started 2024-09-16T10:29:20.798 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:29:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.907+0000 7f943a1c7640 10 monclient: get_monmap_and_config 2024-09-16T10:29:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.907+0000 7f943a1c7640 10 monclient: build_initial_monmap 2024-09-16T10:29:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.907+0000 7f943a1c7640 10 monclient: monmap: 2024-09-16T10:29:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:20.909430+0000 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:20.909430+0000 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:20.912 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:20.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.908+0000 7f943a1c7640 1 Processor -- start 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.908+0000 7f943a1c7640 1 -- start start 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.908+0000 7f943a1c7640 10 monclient: init 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 10 monclient: _add_conns ranks=[4,3,1,7,5,0,2,8,6] 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 10 monclient(hunting): picked mon.noname-c con 0x7f94340a38d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 10 monclient(hunting): picked mon.noname-g con 0x7f943406c670 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 10 monclient(hunting): picked mon.noname-f con 0x7f94340a43c0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9434061ae0 con 0x7f94340a43c0 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94340a04d0 con 0x7f943406c670 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94340a3250 con 0x7f94340a38d0 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f943a1c7640 10 monclient(hunting): authenticate will time out at 728.250168s 2024-09-16T10:29:20.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f94337fe640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f94340a38d0 0x7f94340a3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:59506/0 (socket says 172.21.15.94:59506) 2024-09-16T10:29:20.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.909+0000 7f94337fe640 1 -- 172.21.15.94:0/4243152553 learned_addr learned my addr 172.21.15.94:0/4243152553 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f94337fe640 1 -- 172.21.15.94:0/4243152553 >> v1:172.21.15.94:6791/0 conn(0x7f94340a38d0 legacy=0x7f94340a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f942bfff640 1 -- 172.21.15.94:0/4243152553 >> v1:172.21.15.152:6790/0 conn(0x7f943406c670 legacy=0x7f943413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256023689 0 0) 0x7f94340a3250 con 0x7f94340a38d0 2024-09-16T10:29:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9433fff640 1 -- 172.21.15.94:0/4243152553 >> v1:172.21.15.152:6789/0 conn(0x7f94340a43c0 legacy=0x7f943413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): my global_id is 4144 2024-09-16T10:29:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9420003520 con 0x7f94340a38d0 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631891109 0 0) 0x7f9434061ae0 con 0x7f94340a43c0 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): my global_id is 4150 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9420003840 con 0x7f94340a43c0 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34464848 0 0) 0x7f94340a04d0 con 0x7f943406c670 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient(hunting): my global_id is 4179 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9434061ae0 con 0x7f943406c670 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991620260 0 0) 0x7f9420003520 con 0x7f94340a38d0 2024-09-16T10:29:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 >> v1:172.21.15.152:6790/0 conn(0x7f943406c670 legacy=0x7f943413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 >> v1:172.21.15.152:6789/0 conn(0x7f94340a43c0 legacy=0x7f943413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 monclient: found mon.noname-c 2024-09-16T10:29:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:29:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340a0a00 con 0x7f94340a38d0 2024-09-16T10:29:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f943a1c7640 5 monclient: authenticate success, global_id 4144 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f943a1c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9424001b00 con 0x7f94340a38d0 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient: dump: 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.921 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f943a1c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94240048a0 con 0x7f94340a38d0 2024-09-16T10:29:20.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 1 -- 172.21.15.94:0/4243152553 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9424004d50 con 0x7f94340a38d0 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.911+0000 7f9432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f9432ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f9432ffd640 10 monclient: dump: 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f943a1c7640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f943a1c7640 4 set_mon_vals no callback set 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f943a1c7640 10 monclient: shutdown 2024-09-16T10:29:20.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f943a1c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f943a1c7640 1 -- 172.21.15.94:0/4243152553 >> v1:172.21.15.94:6791/0 conn(0x7f94340a38d0 legacy=0x7f94340a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f943a1c7640 1 -- 172.21.15.94:0/4243152553 shutdown_connections 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f943a1c7640 1 -- 172.21.15.94:0/4243152553 >> 172.21.15.94:0/4243152553 conn(0x7f94340a59d0 msgr2=0x7f94340a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f943a1c7640 1 -- 172.21.15.94:0/4243152553 shutdown_connections 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.912+0000 7f943a1c7640 1 -- 172.21.15.94:0/4243152553 wait complete. 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient: build_initial_monmap 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient: monmap: 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:20.925 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 1 Processor -- start 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 1 -- start start 2024-09-16T10:29:20.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient: init 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient: _add_conns ranks=[3,1,2,8,7,0,6,5,4] 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient(hunting): picked mon.noname-g con 0x7f943406c670 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient(hunting): picked mon.noname-f con 0x7f94340a38d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient(hunting): picked mon.noname-b con 0x7f94340a43c0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943409eff0 con 0x7f94340a38d0 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943409f320 con 0x7f94340a43c0 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943406ba50 con 0x7f943406c670 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.913+0000 7f943a1c7640 10 monclient(hunting): authenticate will time out at 728.254932s 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.914+0000 7f9433fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f94340a43c0 0x7f94341d34e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:46534/0 (socket says 172.21.15.94:46534) 2024-09-16T10:29:20.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.914+0000 7f9433fff640 1 -- 172.21.15.94:0/4005011773 learned_addr learned my addr 172.21.15.94:0/4005011773 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9433fff640 1 -- 172.21.15.94:0/4005011773 >> v1:172.21.15.94:6790/0 conn(0x7f94340a43c0 legacy=0x7f94341d34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f94337fe640 1 -- 172.21.15.94:0/4005011773 >> v1:172.21.15.152:6790/0 conn(0x7f943406c670 legacy=0x7f94341369d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f942bfff640 1 -- 172.21.15.94:0/4005011773 >> v1:172.21.15.152:6789/0 conn(0x7f94340a38d0 legacy=0x7f94341cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337235109 0 0) 0x7f943409f320 con 0x7f94340a43c0 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): my global_id is 4151 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9414003520 con 0x7f94340a43c0 2024-09-16T10:29:20.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091257376 0 0) 0x7f943409eff0 con 0x7f94340a38d0 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): my global_id is 4159 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f943409f320 con 0x7f94340a38d0 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057995412 0 0) 0x7f943406ba50 con 0x7f943406c670 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): my global_id is 4188 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f943409eff0 con 0x7f943406c670 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2390968564 0 0) 0x7f943409f320 con 0x7f94340a38d0 2024-09-16T10:29:20.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f943406ba50 con 0x7f94340a38d0 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f941c001d50 con 0x7f94340a38d0 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): dump: 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.933 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3287186179 0 0) 0x7f9414003520 con 0x7f94340a43c0 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f943409f320 con 0x7f94340a43c0 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f942c0020f0 con 0x7f94340a43c0 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): dump: 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.935 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1422251031 0 0) 0x7f943409eff0 con 0x7f943406c670 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9414003520 con 0x7f943406c670 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9424001ac0 con 0x7f943406c670 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient(hunting): dump: 2024-09-16T10:29:20.936 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.937 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2508244942 0 0) 0x7f943406ba50 con 0x7f94340a38d0 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 >> v1:172.21.15.152:6790/0 conn(0x7f943406c670 legacy=0x7f94341369d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 >> v1:172.21.15.94:6790/0 conn(0x7f94340a43c0 legacy=0x7f94341d34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient: _finish_hunting 0 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 monclient: found mon.f 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94341270e0 con 0x7f94340a38d0 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient: _finish_auth 0 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:20.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f943a1c7640 5 monclient: authenticate success, global_id 4159 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f943a1c7640 10 monclient: _renew_subs 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f943a1c7640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f943a1c7640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94341273c0 con 0x7f94340a38d0 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f943a1c7640 10 monclient: _renew_subs 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f943a1c7640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f943a1c7640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94341256c0 con 0x7f94340a38d0 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f941c0039a0 con 0x7f94340a38d0 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f941c003e50 con 0x7f94340a38d0 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 10 monclient: dump: 2024-09-16T10:29:20.939 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:20.941 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:20.941 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:20.941 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:20.941 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:20.941 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:20.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f941c004370 con 0x7f94340a38d0 2024-09-16T10:29:20.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.916+0000 7f94327fc640 4 set_mon_vals no callback set 2024-09-16T10:29:20.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.917+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2431735127 0 0) 0x7f941c005db0 con 0x7f94340a38d0 2024-09-16T10:29:20.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.917+0000 7f943a1c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:20.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.917+0000 7f943a1c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:20.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.917+0000 7f943a1c7640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:20.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.917+0000 7f943a1c7640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94340008d0 con 0x7f94340a38d0 2024-09-16T10:29:20.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.919+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f941c0057c0 con 0x7f94340a38d0 2024-09-16T10:29:20.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.919+0000 7f9430ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:20.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.919+0000 7f9430ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.974+0000 7f943a1c7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.974+0000 7f943a1c7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.974+0000 7f943a1c7640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.974+0000 7f943a1c7640 1 -- 172.21.15.94:0/4005011773 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94340a16c0 con 0x7f94340a38d0 2024-09-16T10:29:20.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.975+0000 7f9430ff9640 1 -- 172.21.15.94:0/4005011773 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (unknown 150273500 0 1370973882) 0x7f941c005af0 con 0x7f94340a38d0 2024-09-16T10:29:20.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.975+0000 7f9430ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:20.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.975+0000 7f9430ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:20.978 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":8,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","created":"2024-09-16T10:29:16.029056+0000","modified":"2024-09-16T10:29:20.181320+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T10:29:20.181320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19cab4f7-c271-4ce8-895e-75a5871502f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6641fac5-e871-4401-bf30-598b633e70b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6f4cedf2-071d-44f9-87a3-a766668fcd03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3634242e-61e0-4221-8c38-7dce4d5053ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eed750dc-2d29-443f-a91c-d9c5abdcf3d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d6daac7-1ac7-4f0a-b315-27d1333fc3ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:29:20.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.975+0000 7f943a1c7640 10 monclient: shutdown 2024-09-16T10:29:20.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.975+0000 7f943a1c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:20.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.975+0000 7f943a1c7640 1 -- 172.21.15.94:0/4005011773 >> v1:172.21.15.152:6789/0 conn(0x7f94340a38d0 legacy=0x7f94341cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:20.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.975+0000 7f943a1c7640 1 -- 172.21.15.94:0/4005011773 shutdown_connections 2024-09-16T10:29:20.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.975+0000 7f943a1c7640 1 -- 172.21.15.94:0/4005011773 >> 172.21.15.94:0/4005011773 conn(0x7f94340a59d0 msgr2=0x7f94340a7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:20.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.976+0000 7f943a1c7640 1 -- 172.21.15.94:0/4005011773 shutdown_connections 2024-09-16T10:29:20.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:20.976+0000 7f943a1c7640 1 -- 172.21.15.94:0/4005011773 wait complete. 2024-09-16T10:29:20.988 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:20.986+0000 7fb443923040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T10:29:20.989 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T10:29:20.989 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T10:29:21.383 INFO:tasks.ceph.osd.4.smithi152.stderr:2024-09-16T10:29:21.381+0000 7fe1aca11640 -1 Falling back to public interface 2024-09-16T10:29:21.397 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-09-16T10:29:21.394+0000 7f81e9f27640 -1 Falling back to public interface 2024-09-16T10:29:21.401 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-09-16T10:29:21.398+0000 7fb3f0fa8640 -1 Falling back to public interface 2024-09-16T10:29:21.401 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-09-16T10:29:21.398+0000 7f5139a1e640 -1 Falling back to public interface 2024-09-16T10:29:21.404 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T10:29:21.402+0000 7f0bc5862640 -1 Falling back to public interface 2024-09-16T10:29:21.404 INFO:tasks.ceph.osd.5.smithi152.stderr:2024-09-16T10:29:21.402+0000 7f765dbd9640 -1 Falling back to public interface 2024-09-16T10:29:21.594 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T10:29:21.685 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.682+0000 7febff1aa640 10 monclient: get_monmap_and_config 2024-09-16T10:29:21.685 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.682+0000 7febff1aa640 10 monclient: build_initial_monmap 2024-09-16T10:29:21.685 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.682+0000 7febff1aa640 10 monclient: monmap: 2024-09-16T10:29:21.685 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-09-16T10:29:21.685 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:29:21.684728+0000 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:29:21.684728+0000 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:21.686 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:21.687 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.683+0000 7febff1aa640 1 Processor -- start 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.683+0000 7febff1aa640 1 -- start start 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.683+0000 7febff1aa640 10 monclient: init 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 10 monclient: _add_conns ranks=[2,4,3,6,7,1,5,8,0] 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 10 monclient(hunting): picked mon.noname-b con 0x7febf80a2140 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 10 monclient(hunting): picked mon.noname-c con 0x7febf80a15e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 10 monclient(hunting): picked mon.noname-g con 0x7febf80a19f0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf8061ae0 con 0x7febf80a2140 2024-09-16T10:29:21.688 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf80a0920 con 0x7febf80a19f0 2024-09-16T10:29:21.689 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf80a0360 con 0x7febf80a15e0 2024-09-16T10:29:21.689 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:21.689 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febff1aa640 10 monclient(hunting): authenticate will time out at 729.025324s 2024-09-16T10:29:21.689 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febf7fff640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7febf80a15e0 0x7febf813b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:59522/0 (socket says 172.21.15.94:59522) 2024-09-16T10:29:21.689 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febfcf46640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7febf80a2140 0x7febf80a2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:46548/0 (socket says 172.21.15.94:46548) 2024-09-16T10:29:21.689 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.684+0000 7febf7fff640 1 -- 172.21.15.94:0/983177793 learned_addr learned my addr 172.21.15.94:0/983177793 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:21.692 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf7fff640 1 -- 172.21.15.94:0/983177793 >> v1:172.21.15.94:6791/0 conn(0x7febf80a15e0 legacy=0x7febf813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febfcf46640 1 -- 172.21.15.94:0/983177793 >> v1:172.21.15.94:6790/0 conn(0x7febf80a2140 legacy=0x7febf80a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276028742 0 0) 0x7febf80a0360 con 0x7febf80a15e0 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): my global_id is 4153 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febfd747640 1 -- 172.21.15.94:0/983177793 >> v1:172.21.15.152:6790/0 conn(0x7febf80a19f0 legacy=0x7febf813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe8003520 con 0x7febf80a15e0 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704980602 0 0) 0x7febf8061ae0 con 0x7febf80a2140 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:21.693 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): my global_id is 4160 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf80a0360 con 0x7febf80a2140 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331670572 0 0) 0x7febf80a0920 con 0x7febf80a19f0 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 10 monclient(hunting): my global_id is 4206 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.686+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf8061ae0 con 0x7febf80a19f0 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1733542897 0 0) 0x7febf80a0360 con 0x7febf80a2140 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 >> v1:172.21.15.94:6791/0 conn(0x7febf80a15e0 legacy=0x7febf813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 >> v1:172.21.15.152:6790/0 conn(0x7febf80a19f0 legacy=0x7febf813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:21.694 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 1 monclient: found mon.noname-b 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80a0640 con 0x7febf80a2140 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 5 monclient: authenticate success, global_id 4160 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7febec001a60 con 0x7febf80a2140 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:21.695 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: dump: 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:21.696 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febec001d70 con 0x7febf80a2140 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 1 -- 172.21.15.94:0/983177793 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7febec004b20 con 0x7febf80a2140 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 4 set_mon_vals no callback set 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febf77fe640 10 monclient: dump: 2024-09-16T10:29:21.697 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:21.698 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 10 monclient: shutdown 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 1 -- 172.21.15.94:0/983177793 >> v1:172.21.15.94:6790/0 conn(0x7febf80a2140 legacy=0x7febf80a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 1 -- 172.21.15.94:0/983177793 shutdown_connections 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 1 -- 172.21.15.94:0/983177793 >> 172.21.15.94:0/983177793 conn(0x7febf80a93c0 msgr2=0x7febf80a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 1 -- 172.21.15.94:0/983177793 shutdown_connections 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.687+0000 7febff1aa640 1 -- 172.21.15.94:0/983177793 wait complete. 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.688+0000 7febff1aa640 10 monclient: build_initial_monmap 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.688+0000 7febff1aa640 10 monclient: monmap: 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:21.699 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:21.700 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.688+0000 7febff1aa640 1 Processor -- start 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.688+0000 7febff1aa640 1 -- start start 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.688+0000 7febff1aa640 10 monclient: init 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 10 monclient: _add_conns ranks=[3,8,5,1,4,6,0,7,2] 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 10 monclient(hunting): picked mon.noname-g con 0x7febf80a15e0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 10 monclient(hunting): picked mon.noname-e con 0x7febf80a19f0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 10 monclient(hunting): picked mon.noname-h con 0x7febf80a2140 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf81213e0 con 0x7febf80a15e0 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf811f380 con 0x7febf80a2140 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf80a7990 con 0x7febf80a19f0 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:21.701 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febff1aa640 10 monclient(hunting): authenticate will time out at 729.030348s 2024-09-16T10:29:21.702 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febf7fff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7febf80a19f0 0x7febf813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:45264/0 (socket says 172.21.15.94:45264) 2024-09-16T10:29:21.702 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.689+0000 7febf7fff640 1 -- 172.21.15.94:0/2645479741 learned_addr learned my addr 172.21.15.94:0/2645479741 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:21.706 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.691+0000 7febf7fff640 1 -- 172.21.15.94:0/2645479741 >> v1:172.21.15.94:6793/0 conn(0x7febf80a19f0 legacy=0x7febf813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:21.706 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.691+0000 7febfd747640 1 -- 172.21.15.94:0/2645479741 >> v1:172.21.15.152:6791/0 conn(0x7febf80a2140 legacy=0x7febf813f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:21.706 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.691+0000 7febfcf46640 1 -- 172.21.15.94:0/2645479741 >> v1:172.21.15.152:6790/0 conn(0x7febf80a15e0 legacy=0x7febf813e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:21.706 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.691+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867763738 0 0) 0x7febf80a7990 con 0x7febf80a19f0 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.691+0000 7febf57fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.691+0000 7febf57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.691+0000 7febf57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.691+0000 7febf57fa640 10 monclient(hunting): my global_id is 4193 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.691+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc003520 con 0x7febf80a19f0 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906501846 0 0) 0x7febf81213e0 con 0x7febf80a15e0 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): my global_id is 4215 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf80a7990 con 0x7febf80a15e0 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576612130 0 0) 0x7febf811f380 con 0x7febf80a2140 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:21.707 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): my global_id is 4172 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf81213e0 con 0x7febf80a2140 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2981290640 0 0) 0x7febdc003520 con 0x7febf80a19f0 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febf811f380 con 0x7febf80a19f0 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7febe4001920 con 0x7febf80a19f0 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): dump: 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.708 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:21.709 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2178382022 0 0) 0x7febf80a7990 con 0x7febf80a15e0 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febdc003520 con 0x7febf80a15e0 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7febec004b20 con 0x7febf80a15e0 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): dump: 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:21.710 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1113197233 0 0) 0x7febf81213e0 con 0x7febf80a2140 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febf80a7990 con 0x7febf80a2140 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7febf00020d0 con 0x7febf80a2140 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:21.711 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient(hunting): dump: 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:21.712 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1418406310 0 0) 0x7febf811f380 con 0x7febf80a19f0 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 >> v1:172.21.15.152:6791/0 conn(0x7febf80a2140 legacy=0x7febf813f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 >> v1:172.21.15.152:6790/0 conn(0x7febf80a15e0 legacy=0x7febf813e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 monclient: found mon.e 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80a7440 con 0x7febf80a19f0 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:21.713 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febff1aa640 5 monclient: authenticate success, global_id 4193 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febff1aa640 10 monclient: _renew_subs 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febff1aa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febff1aa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.94:6793/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf809f7b0 con 0x7febf80a19f0 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febff1aa640 10 monclient: _renew_subs 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febff1aa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febff1aa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febf811fc80 con 0x7febf80a19f0 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febe4004dd0 con 0x7febf80a19f0 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7febe4005280 con 0x7febf80a19f0 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:21.714 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf57fa640 10 monclient: dump: 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:21.715 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.692+0000 7febf6ffd640 4 set_mon_vals no callback set 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.693+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 7 ==== mgrmap(e 2) v1 ==== 44528+0+0 (unknown 1348615700 0 0) 0x7febe40101e0 con 0x7febf80a19f0 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.693+0000 7febff1aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.693+0000 7febff1aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.693+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2431735127 0 0) 0x7febe4045190 con 0x7febf80a19f0 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.693+0000 7febff1aa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.693+0000 7febff1aa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.94:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8120e20 con 0x7febf80a19f0 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.695+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7febe40150c0 con 0x7febf80a19f0 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.695+0000 7febf57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:21.716 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.695+0000 7febf57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:21.752 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.749+0000 7febff1aa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:21.752 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.749+0000 7febff1aa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:21.752 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.749+0000 7febff1aa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:21.752 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.749+0000 7febff1aa640 1 -- 172.21.15.94:0/2645479741 --> v1:172.21.15.94:6793/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febf813fcd0 con 0x7febf80a19f0 2024-09-16T10:29:21.753 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.750+0000 7febf57fa640 1 -- 172.21.15.94:0/2645479741 <== mon.8 v1:172.21.15.94:6793/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (unknown 150273500 0 1370973882) 0x7febe4014ee0 con 0x7febf80a19f0 2024-09-16T10:29:21.753 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.750+0000 7febf57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:21.753 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.750+0000 7febf57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:21.753 INFO:teuthology.misc.health.smithi094.stdout: 2024-09-16T10:29:21.754 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":8,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","created":"2024-09-16T10:29:16.029056+0000","modified":"2024-09-16T10:29:20.181320+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T10:29:20.181320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19cab4f7-c271-4ce8-895e-75a5871502f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6641fac5-e871-4401-bf30-598b633e70b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6f4cedf2-071d-44f9-87a3-a766668fcd03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3634242e-61e0-4221-8c38-7dce4d5053ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eed750dc-2d29-443f-a91c-d9c5abdcf3d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d6daac7-1ac7-4f0a-b315-27d1333fc3ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:29:21.754 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.751+0000 7febff1aa640 10 monclient: shutdown 2024-09-16T10:29:21.755 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.751+0000 7febff1aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:21.755 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.751+0000 7febff1aa640 1 -- 172.21.15.94:0/2645479741 >> v1:172.21.15.94:6793/0 conn(0x7febf80a19f0 legacy=0x7febf813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:21.755 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.751+0000 7febff1aa640 1 -- 172.21.15.94:0/2645479741 shutdown_connections 2024-09-16T10:29:21.755 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.751+0000 7febff1aa640 1 -- 172.21.15.94:0/2645479741 >> 172.21.15.94:0/2645479741 conn(0x7febf80a93c0 msgr2=0x7febf80a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:21.755 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.751+0000 7febff1aa640 1 -- 172.21.15.94:0/2645479741 shutdown_connections 2024-09-16T10:29:21.755 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:21.751+0000 7febff1aa640 1 -- 172.21.15.94:0/2645479741 wait complete. 2024-09-16T10:29:21.762 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T10:29:22.025 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:22.023+0000 7fb3e4318640 -1 mgr.server handle_report got status from non-daemon mon.i 2024-09-16T10:29:22.025 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:22.023+0000 7fb3e4318640 -1 mgr.server handle_report got status from non-daemon mon.f 2024-09-16T10:29:22.026 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:22.024+0000 7fb3e4318640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T10:29:22.026 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:22.024+0000 7fb3e4318640 -1 mgr.server handle_report got status from non-daemon mon.d 2024-09-16T10:29:22.026 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:22.025+0000 7fb3e4318640 -1 mgr.server handle_report got status from non-daemon mon.g 2024-09-16T10:29:22.028 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:22.026+0000 7fb3e4318640 -1 mgr.server handle_report got status from non-daemon mon.h 2024-09-16T10:29:22.028 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:22.026+0000 7fb3e4318640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T10:29:22.028 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T10:29:22.026+0000 7fb3e4318640 -1 mgr.server handle_report got status from non-daemon mon.e 2024-09-16T10:29:24.804 INFO:tasks.ceph.osd.4.smithi152.stderr:2024-09-16T10:29:24.802+0000 7fe1aca11640 -1 osd.4 0 log_to_monitors true 2024-09-16T10:29:24.849 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T10:29:24.847+0000 7f0bc5862640 -1 osd.3 0 log_to_monitors true 2024-09-16T10:29:24.864 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-09-16T10:29:24.861+0000 7fb3f0fa8640 -1 osd.0 0 log_to_monitors true 2024-09-16T10:29:24.892 INFO:tasks.ceph.osd.5.smithi152.stderr:2024-09-16T10:29:24.890+0000 7f765dbd9640 -1 osd.5 0 log_to_monitors true 2024-09-16T10:29:24.893 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-09-16T10:29:24.890+0000 7f81e9f27640 -1 osd.1 0 log_to_monitors true 2024-09-16T10:29:24.917 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-09-16T10:29:24.915+0000 7f5139a1e640 -1 osd.2 0 log_to_monitors true 2024-09-16T10:29:26.408 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T10:29:26.406+0000 7f0bc0fcc640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T10:29:26.409 INFO:tasks.ceph.osd.4.smithi152.stderr:2024-09-16T10:29:26.406+0000 7fe1a7b6d640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T10:29:26.409 INFO:tasks.ceph.osd.5.smithi152.stderr:2024-09-16T10:29:26.407+0000 7f7659343640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T10:29:26.414 INFO:tasks.ceph.osd.4.smithi152.stderr:2024-09-16T10:29:26.412+0000 7fe1a41a8640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:29:26.414 INFO:tasks.ceph.osd.5.smithi152.stderr:2024-09-16T10:29:26.412+0000 7f765597e640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:29:26.415 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T10:29:26.413+0000 7f0bbd607640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:29:26.881 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-09-16T10:29:26.879+0000 7f81e6ecc640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T10:29:26.889 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-09-16T10:29:26.887+0000 7f81ddcc4640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:29:26.894 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-09-16T10:29:26.891+0000 7fb3ec712640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T10:29:26.900 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-09-16T10:29:26.898+0000 7fb3e4d45640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:29:27.598 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-09-16T10:29:27.596+0000 7f5136326640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T10:29:27.604 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-09-16T10:29:27.602+0000 7f512d11e640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:29:28.367 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T10:29:28.463 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.461+0000 7f0541776640 10 monclient: get_monmap_and_config 2024-09-16T10:29:28.463 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.461+0000 7f0541776640 10 monclient: build_initial_monmap 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.461+0000 7f0541776640 10 monclient: monmap: 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:29:28.462239+0000 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:29:28.462239+0000 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:28.464 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:28.465 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:28.465 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:28.465 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:28.465 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:28.465 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:28.465 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:28.465 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 1 Processor -- start 2024-09-16T10:29:28.465 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 1 -- start start 2024-09-16T10:29:28.465 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 10 monclient: init 2024-09-16T10:29:28.470 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:28.471 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 10 monclient: _add_conns ranks=[0,5,4,3,1,7,6,8,2] 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 10 monclient(hunting): picked mon.noname-a con 0x7f053c0a2140 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 10 monclient(hunting): picked mon.noname-h con 0x7f053c0a15e0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 10 monclient(hunting): picked mon.noname-c con 0x7f053c0a19f0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c061ae0 con 0x7f053c0a2140 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c0a0920 con 0x7f053c0a19f0 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c0a0360 con 0x7f053c0a15e0 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f0541776640 10 monclient(hunting): authenticate will time out at 735.802808s 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f053affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f053c0a2140 0x7f053c0a2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59308/0 (socket says 172.21.15.94:59308) 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f053affd640 1 -- 172.21.15.94:0/2313613102 learned_addr learned my addr 172.21.15.94:0/2313613102 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.462+0000 7f053b7fe640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f053c0a19f0 0x7f053c13ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33900/0 (socket says 172.21.15.94:33900) 2024-09-16T10:29:28.472 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053b7fe640 1 -- 172.21.15.94:0/2313613102 >> v1:172.21.15.94:6791/0 conn(0x7f053c0a19f0 legacy=0x7f053c13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053affd640 1 -- 172.21.15.94:0/2313613102 >> v1:172.21.15.94:6789/0 conn(0x7f053c0a2140 legacy=0x7f053c0a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156824046 0 0) 0x7f053c0a0920 con 0x7f053c0a19f0 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient(hunting): my global_id is 4162 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0528003520 con 0x7f053c0a19f0 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461711984 0 0) 0x7f053c061ae0 con 0x7f053c0a2140 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:28.473 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient(hunting): my global_id is 4212 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053c0a0920 con 0x7f053c0a2140 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f0533fff640 1 -- 172.21.15.94:0/2313613102 >> v1:172.21.15.152:6791/0 conn(0x7f053c0a15e0 legacy=0x7f053c13b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 651871857 0 0) 0x7f0528003520 con 0x7f053c0a19f0 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 >> v1:172.21.15.152:6791/0 conn(0x7f053c0a15e0 legacy=0x7f053c13b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 >> v1:172.21.15.94:6789/0 conn(0x7f053c0a2140 legacy=0x7f053c0a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 monclient: found mon.noname-c 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c0a0640 con 0x7f053c0a19f0 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:28.474 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f0541776640 5 monclient: authenticate success, global_id 4162 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0534001de0 con 0x7f053c0a19f0 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: dump: 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:28.475 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f0541776640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0534004b60 con 0x7f053c0a19f0 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:28.476 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 1 -- 172.21.15.94:0/2313613102 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0534002100 con 0x7f053c0a19f0 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f0541776640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f0541776640 4 set_mon_vals no callback set 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.465+0000 7f053a7fc640 10 monclient: dump: 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:28.477 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.466+0000 7f0541776640 10 monclient: shutdown 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.466+0000 7f0541776640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.466+0000 7f0541776640 1 -- 172.21.15.94:0/2313613102 >> v1:172.21.15.94:6791/0 conn(0x7f053c0a19f0 legacy=0x7f053c13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.466+0000 7f0541776640 1 -- 172.21.15.94:0/2313613102 shutdown_connections 2024-09-16T10:29:28.478 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.466+0000 7f0541776640 1 -- 172.21.15.94:0/2313613102 >> 172.21.15.94:0/2313613102 conn(0x7f053c0a93c0 msgr2=0x7f053c0a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.466+0000 7f0541776640 1 -- 172.21.15.94:0/2313613102 shutdown_connections 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.466+0000 7f0541776640 1 -- 172.21.15.94:0/2313613102 wait complete. 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.466+0000 7f0541776640 10 monclient: build_initial_monmap 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.466+0000 7f0541776640 10 monclient: monmap: 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:28.479 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 1 Processor -- start 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 1 -- start start 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 10 monclient: init 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 10 monclient: _add_conns ranks=[0,4,7,1,5,6,2,8,3] 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 10 monclient(hunting): picked mon.noname-a con 0x7f053c0a15e0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:28.480 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 10 monclient(hunting): picked mon.noname-c con 0x7f053c0a19f0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:28.481 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 10 monclient(hunting): picked mon.noname-i con 0x7f053c0a2140 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:28.481 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c139300 con 0x7f053c0a15e0 2024-09-16T10:29:28.481 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c09f760 con 0x7f053c0a19f0 2024-09-16T10:29:28.481 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c120980 con 0x7f053c0a2140 2024-09-16T10:29:28.481 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f053affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f053c0a15e0 0x7f053c1cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59322/0 (socket says 172.21.15.94:59322) 2024-09-16T10:29:28.481 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f053affd640 1 -- 172.21.15.94:0/3994640922 learned_addr learned my addr 172.21.15.94:0/3994640922 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:28.481 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:28.481 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.467+0000 7f0541776640 10 monclient(hunting): authenticate will time out at 735.807891s 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f053affd640 1 -- 172.21.15.94:0/3994640922 >> v1:172.21.15.94:6789/0 conn(0x7f053c0a15e0 legacy=0x7f053c1cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f0533fff640 1 -- 172.21.15.94:0/3994640922 >> v1:172.21.15.94:6791/0 conn(0x7f053c0a19f0 legacy=0x7f053c1cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909789971 0 0) 0x7f053c139300 con 0x7f053c0a15e0 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): my global_id is 4221 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f051c003520 con 0x7f053c0a15e0 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262836845 0 0) 0x7f053c09f760 con 0x7f053c0a19f0 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:28.485 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): my global_id is 4171 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053c139300 con 0x7f053c0a19f0 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f053b7fe640 1 -- 172.21.15.94:0/3994640922 >> v1:172.21.15.152:6792/0 conn(0x7f053c0a2140 legacy=0x7f053c1d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1006280294 0 0) 0x7f051c003520 con 0x7f053c0a15e0 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f053c09f760 con 0x7f053c0a15e0 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f052c001cb0 con 0x7f053c0a15e0 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): dump: 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:28.486 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:28.487 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1887938843 0 0) 0x7f053c139300 con 0x7f053c0a19f0 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f051c003520 con 0x7f053c0a19f0 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0524001cd0 con 0x7f053c0a19f0 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): dump: 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:28.488 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116791674 0 0) 0x7f053c120980 con 0x7f053c0a2140 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:28.489 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient(hunting): my global_id is 4237 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053c139300 con 0x7f053c0a2140 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1536392239 0 0) 0x7f053c09f760 con 0x7f053c0a15e0 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 >> v1:172.21.15.152:6792/0 conn(0x7f053c0a2140 legacy=0x7f053c1d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 >> v1:172.21.15.94:6791/0 conn(0x7f053c0a19f0 legacy=0x7f053c1cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 monclient: found mon.a 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c1cbf90 con 0x7f053c0a15e0 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:28.490 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f0541776640 5 monclient: authenticate success, global_id 4221 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f0541776640 10 monclient: _renew_subs 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f0541776640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c0a79c0 con 0x7f053c0a15e0 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f0541776640 10 monclient: _renew_subs 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f0541776640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c1206a0 con 0x7f053c0a15e0 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f052c004dd0 con 0x7f053c0a15e0 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f052c005280 con 0x7f053c0a15e0 2024-09-16T10:29:28.491 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f05337fe640 10 monclient: dump: 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:28.492 INFO:teuthology.misc.health.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr: 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.470+0000 7f0539ffb640 4 set_mon_vals no callback set 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.471+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 44726+0+0 (unknown 1094748937 0 0) 0x7f052c0102a0 con 0x7f053c0a15e0 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.471+0000 7f0541776640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.471+0000 7f0541776640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.471+0000 7f0541776640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.471+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c1202b0 con 0x7f053c0a15e0 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.471+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 3142+0+0 (unknown 4202137849 0 0) 0x7f052c045cd0 con 0x7f053c0a15e0 2024-09-16T10:29:28.493 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.472+0000 7f0533fff640 10 monclient: get_auth_request con 0x7f051c0397e0 auth_method 0 2024-09-16T10:29:28.494 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.476+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f052c015430 con 0x7f053c0a15e0 2024-09-16T10:29:28.494 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.476+0000 7f05337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:28.494 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.476+0000 7f05337fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.641+0000 7f0541776640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.641+0000 7f0541776640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.641+0000 7f0541776640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.641+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f053c0a4b40 con 0x7f053c0a15e0 2024-09-16T10:29:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.642+0000 7f05337fe640 1 -- 172.21.15.94:0/3994640922 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+8964 (unknown 4250787923 0 2666841193) 0x7f052c0138a0 con 0x7f053c0a15e0 2024-09-16T10:29:28.644 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.642+0000 7f05337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:28.644 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.642+0000 7f05337fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:28.644 INFO:teuthology.misc.health.smithi094.stdout: 2024-09-16T10:29:28.644 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":12,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","created":"2024-09-16T10:29:16.029056+0000","modified":"2024-09-16T10:29:27.994140+0000","last_up_change":"2024-09-16T10:29:27.994140+0000","last_in_change":"2024-09-16T10:29:20.181320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:29:27.046989+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"19cab4f7-c271-4ce8-895e-75a5871502f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":4202994153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":4202994153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":4202994153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":4202994153}]},"public_addr":"172.21.15.94:6804/4202994153","cluster_addr":"172.21.15.94:6806/4202994153","heartbeat_back_addr":"172.21.15.94:6810/4202994153","heartbeat_front_addr":"172.21.15.94:6808/4202994153","state":["exists","up"]},{"osd":1,"uuid":"6641fac5-e871-4401-bf30-598b633e70b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":782646670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":782646670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":782646670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":782646670}]},"public_addr":"172.21.15.94:6800/782646670","cluster_addr":"172.21.15.94:6801/782646670","heartbeat_back_addr":"172.21.15.94:6803/782646670","heartbeat_front_addr":"172.21.15.94:6802/782646670","state":["exists","up"]},{"osd":2,"uuid":"6f4cedf2-071d-44f9-87a3-a766668fcd03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3102113414}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3102113414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3102113414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3102113414}]},"public_addr":"172.21.15.94:6805/3102113414","cluster_addr":"172.21.15.94:6807/3102113414","heartbeat_back_addr":"172.21.15.94:6811/3102113414","heartbeat_front_addr":"172.21.15.94:6809/3102113414","state":["exists","up"]},{"osd":3,"uuid":"3634242e-61e0-4221-8c38-7dce4d5053ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6805","nonce":823649059}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6807","nonce":823649059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6811","nonce":823649059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6809","nonce":823649059}]},"public_addr":"172.21.15.152:6805/823649059","cluster_addr":"172.21.15.152:6807/823649059","heartbeat_back_addr":"172.21.15.152:6811/823649059","heartbeat_front_addr":"172.21.15.152:6809/823649059","state":["exists","up"]},{"osd":4,"uuid":"eed750dc-2d29-443f-a91c-d9c5abdcf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6801","nonce":1475699231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6802","nonce":1475699231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6804","nonce":1475699231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6803","nonce":1475699231}]},"public_addr":"172.21.15.152:6801/1475699231","cluster_addr":"172.21.15.152:6802/1475699231","heartbeat_back_addr":"172.21.15.152:6804/1475699231","heartbeat_front_addr":"172.21.15.152:6803/1475699231","state":["exists","up"]},{"osd":5,"uuid":"5d6daac7-1ac7-4f0a-b315-27d1333fc3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6806","nonce":3657457384}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6808","nonce":3657457384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6812","nonce":3657457384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6810","nonce":3657457384}]},"public_addr":"172.21.15.152:6806/3657457384","cluster_addr":"172.21.15.152:6808/3657457384","heartbeat_back_addr":"172.21.15.152:6812/3657457384","heartbeat_front_addr":"172.21.15.152:6810/3657457384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.818679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.874108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.872051+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.780789+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.901652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:29:28.646 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.644+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 >> v1:172.21.15.152:6800/2686360781 conn(0x7f051c0397e0 legacy=0x7f051c03bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.646 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.644+0000 7f0541776640 10 monclient: shutdown 2024-09-16T10:29:28.646 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.644+0000 7f0541776640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:28.646 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.644+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 >> v1:172.21.15.94:6789/0 conn(0x7f053c0a15e0 legacy=0x7f053c1cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.646 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.644+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 shutdown_connections 2024-09-16T10:29:28.646 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.644+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 >> 172.21.15.94:0/3994640922 conn(0x7f053c0a93c0 msgr2=0x7f053c0a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:28.646 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.644+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 shutdown_connections 2024-09-16T10:29:28.646 INFO:teuthology.misc.health.smithi094.stderr:2024-09-16T10:29:28.644+0000 7f0541776640 1 -- 172.21.15.94:0/3994640922 wait complete. 2024-09-16T10:29:28.655 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T10:29:28.655 INFO:tasks.ceph:Creating RBD pool 2024-09-16T10:29:28.655 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T10:29:28.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.749+0000 7f8d579bc640 10 monclient: get_monmap_and_config 2024-09-16T10:29:28.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.749+0000 7f8d579bc640 10 monclient: build_initial_monmap 2024-09-16T10:29:28.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.749+0000 7f8d579bc640 10 monclient: monmap: 2024-09-16T10:29:28.751 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:28.751 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.751 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:28.750396+0000 2024-09-16T10:29:28.751 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:28.750396+0000 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:28.752 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 1 Processor -- start 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 1 -- start start 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 10 monclient: init 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 10 monclient: _add_conns ranks=[2,3,6,4,5,8,7,0,1] 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 10 monclient(hunting): picked mon.noname-b con 0x7f8d500a3860 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 10 monclient(hunting): picked mon.noname-g con 0x7f8d5006c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 10 monclient(hunting): picked mon.noname-d con 0x7f8d500a4350 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d50061ae0 con 0x7f8d500a3860 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d500a0600 con 0x7f8d5006c1d0 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d500a0b30 con 0x7f8d500a4350 2024-09-16T10:29:28.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:28.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d579bc640 10 monclient(hunting): authenticate will time out at 736.090808s 2024-09-16T10:29:28.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d55758640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f8d500a3860 0x7f8d500a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53350/0 (socket says 172.21.15.94:53350) 2024-09-16T10:29:28.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d55f59640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f8d500a4350 0x7f8d5013ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:44252/0 (socket says 172.21.15.94:44252) 2024-09-16T10:29:28.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.750+0000 7f8d55758640 1 -- 172.21.15.94:0/3669002553 learned_addr learned my addr 172.21.15.94:0/3669002553 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:28.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d55f59640 1 -- 172.21.15.94:0/3669002553 >> v1:172.21.15.94:6792/0 conn(0x7f8d500a4350 legacy=0x7f8d5013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d55758640 1 -- 172.21.15.94:0/3669002553 >> v1:172.21.15.94:6790/0 conn(0x7f8d500a3860 legacy=0x7f8d500a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174510673 0 0) 0x7f8d500a0b30 con 0x7f8d500a4350 2024-09-16T10:29:28.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:28.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:28.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:28.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient(hunting): my global_id is 4146 2024-09-16T10:29:28.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c003520 con 0x7f8d500a4350 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983550194 0 0) 0x7f8d50061ae0 con 0x7f8d500a3860 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient(hunting): my global_id is 4205 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d500a0b30 con 0x7f8d500a3860 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d54f57640 1 -- 172.21.15.94:0/3669002553 >> v1:172.21.15.152:6790/0 conn(0x7f8d5006c1d0 legacy=0x7f8d5013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 315386343 0 0) 0x7f8d3c003520 con 0x7f8d500a4350 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 >> v1:172.21.15.152:6790/0 conn(0x7f8d5006c1d0 legacy=0x7f8d5013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 >> v1:172.21.15.94:6790/0 conn(0x7f8d500a3860 legacy=0x7f8d500a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:28.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 monclient: found mon.noname-d 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d500a0e10 con 0x7f8d500a4350 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: _finish_auth 0 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d579bc640 5 monclient: authenticate success, global_id 4146 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8d48001de0 con 0x7f8d500a4350 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d579bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: dump: 2024-09-16T10:29:28.760 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:28.761 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d579bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d480020f0 con 0x7f8d500a4350 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 1 -- 172.21.15.94:0/3669002553 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8d48004ea0 con 0x7f8d500a4350 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d579bc640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d579bc640 4 set_mon_vals no callback set 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.753+0000 7f8d47fff640 10 monclient: dump: 2024-09-16T10:29:28.762 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:28.763 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.754+0000 7f8d579bc640 10 monclient: shutdown 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.754+0000 7f8d579bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.754+0000 7f8d579bc640 1 -- 172.21.15.94:0/3669002553 >> v1:172.21.15.94:6792/0 conn(0x7f8d500a4350 legacy=0x7f8d5013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.754+0000 7f8d579bc640 1 -- 172.21.15.94:0/3669002553 shutdown_connections 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.754+0000 7f8d579bc640 1 -- 172.21.15.94:0/3669002553 >> 172.21.15.94:0/3669002553 conn(0x7f8d500a5960 msgr2=0x7f8d500a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.754+0000 7f8d579bc640 1 -- 172.21.15.94:0/3669002553 shutdown_connections 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.754+0000 7f8d579bc640 1 -- 172.21.15.94:0/3669002553 wait complete. 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.754+0000 7f8d579bc640 10 monclient: build_initial_monmap 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.754+0000 7f8d579bc640 10 monclient: monmap: 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:28.764 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 1 Processor -- start 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 1 -- start start 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 10 monclient: init 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 10 monclient: _add_conns ranks=[6,0,7,5,4,1,3,2,8] 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 10 monclient(hunting): picked mon.noname-d con 0x7f8d5006c1d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 10 monclient(hunting): picked mon.noname-a con 0x7f8d500a3860 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 10 monclient(hunting): picked mon.noname-i con 0x7f8d500a4350 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5009f530 con 0x7f8d500a3860 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5009f070 con 0x7f8d5006c1d0 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d501abc70 con 0x7f8d500a4350 2024-09-16T10:29:28.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:28.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d579bc640 10 monclient(hunting): authenticate will time out at 736.095644s 2024-09-16T10:29:28.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d55758640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f8d5006c1d0 0x7f8d501be590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:44268/0 (socket says 172.21.15.94:44268) 2024-09-16T10:29:28.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.755+0000 7f8d55758640 1 -- 172.21.15.94:0/490694484 learned_addr learned my addr 172.21.15.94:0/490694484 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.757+0000 7f8d55758640 1 -- 172.21.15.94:0/490694484 >> v1:172.21.15.94:6792/0 conn(0x7f8d5006c1d0 legacy=0x7f8d501be590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.757+0000 7f8d54f57640 1 -- 172.21.15.94:0/490694484 >> v1:172.21.15.94:6789/0 conn(0x7f8d500a3860 legacy=0x7f8d501cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.757+0000 7f8d55758640 1 --1- 172.21.15.94:0/490694484 >> v1:172.21.15.94:6792/0 conn(0x7f8d5006c1d0 0x7f8d501be590 :-1 s=OPENED pgs=54 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7f8d5009f070 auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.757+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319509361 0 0) 0x7f8d5009f530 con 0x7f8d500a3860 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.757+0000 7f8d45ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.757+0000 7f8d45ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.757+0000 7f8d45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.757+0000 7f8d45ffb640 10 monclient(hunting): my global_id is 4230 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.757+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d34003550 con 0x7f8d500a3860 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d55f59640 1 -- 172.21.15.94:0/490694484 >> v1:172.21.15.152:6792/0 conn(0x7f8d500a4350 legacy=0x7f8d501d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:28.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 310051525 0 0) 0x7f8d34003550 con 0x7f8d500a3860 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d5009f530 con 0x7f8d500a3860 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8d40001bf0 con 0x7f8d500a3860 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient(hunting): dump: 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:28.771 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493562242 0 0) 0x7f8d501abc70 con 0x7f8d500a4350 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient(hunting): my global_id is 4246 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d34003550 con 0x7f8d500a4350 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1132372763 0 0) 0x7f8d5009f530 con 0x7f8d500a3860 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 >> v1:172.21.15.152:6792/0 conn(0x7f8d500a4350 legacy=0x7f8d501d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 >> v1:172.21.15.94:6792/0 conn(0x7f8d5006c1d0 legacy=0x7f8d501be590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 monclient: found mon.a 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5006b9a0 con 0x7f8d500a3860 2024-09-16T10:29:28.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient: _finish_auth 0 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d579bc640 5 monclient: authenticate success, global_id 4230 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884042112 0 0) 0x7f8d5009f070 con 0x7f8d5006c1d0 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d579bc640 10 monclient: _renew_subs 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d579bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d501aa750 con 0x7f8d500a3860 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d40003a80 con 0x7f8d500a3860 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8d40003f30 con 0x7f8d500a3860 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d45ffb640 10 monclient: dump: 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:28.775 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d477fe640 4 set_mon_vals no callback set 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d579bc640 10 monclient: _renew_subs 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d579bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.758+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d501beca0 con 0x7f8d500a3860 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.759+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 44726+0+0 (unknown 1094748937 0 0) 0x7f8d40004090 con 0x7f8d500a3860 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.759+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 3142+0+0 (unknown 4202137849 0 0) 0x7f8d40045b90 con 0x7f8d500a3860 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.759+0000 7f8d579bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.759+0000 7f8d579bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.759+0000 7f8d579bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.759+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d501beef0 con 0x7f8d500a3860 2024-09-16T10:29:28.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.759+0000 7f8d55758640 10 monclient: get_auth_request con 0x7f8d34038fa0 auth_method 0 2024-09-16T10:29:28.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.764+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f8d400150c0 con 0x7f8d500a3860 2024-09-16T10:29:28.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.764+0000 7f8d45ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:28.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.764+0000 7f8d45ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:28.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.925+0000 7f8d579bc640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T10:29:28.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.925+0000 7f8d579bc640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T10:29:28.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.925+0000 7f8d579bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:28.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:28.925+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f8d500665f0 con 0x7f8d500a3860 2024-09-16T10:29:29.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.001+0000 7f8d45ffb640 1 -- 172.21.15.94:0/490694484 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (unknown 2984082091 0 0) 0x7f8d40013760 con 0x7f8d500a3860 2024-09-16T10:29:29.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.001+0000 7f8d45ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T10:29:29.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.001+0000 7f8d45ffb640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T10:29:29.003 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'rbd' created 2024-09-16T10:29:29.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.004+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8d34038fa0 legacy=0x7f8d3403b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:29.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.004+0000 7f8d579bc640 10 monclient: shutdown 2024-09-16T10:29:29.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.004+0000 7f8d579bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:29.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.004+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 >> v1:172.21.15.94:6789/0 conn(0x7f8d500a3860 legacy=0x7f8d501cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:29.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.004+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 shutdown_connections 2024-09-16T10:29:29.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.004+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 >> 172.21.15.94:0/490694484 conn(0x7f8d500a5960 msgr2=0x7f8d500a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:29.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.004+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 shutdown_connections 2024-09-16T10:29:29.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.004+0000 7f8d579bc640 1 -- 172.21.15.94:0/490694484 wait complete. 2024-09-16T10:29:29.017 DEBUG:teuthology.orchestra.run.smithi094:> rbd --cluster ceph pool init rbd 2024-09-16T10:29:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.311+0000 7f392b26ec00 10 monclient: get_monmap_and_config 2024-09-16T10:29:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.311+0000 7f392b26ec00 10 monclient: build_initial_monmap 2024-09-16T10:29:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.311+0000 7f392b26ec00 10 monclient: monmap: 2024-09-16T10:29:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:29.312380+0000 2024-09-16T10:29:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:29.312380+0000 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:29.314 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 1 Processor -- start 2024-09-16T10:29:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 1 -- start start 2024-09-16T10:29:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 10 monclient: init 2024-09-16T10:29:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 10 monclient: _reopen_session rank -1 2024-09-16T10:29:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 10 monclient: _add_conns ranks=[8,4,1,6,5,0,3,2,7] 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 10 monclient(hunting): picked mon.noname-e con 0x55bb6c123630 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 10 monclient(hunting): picked mon.noname-c con 0x55bb6c123a10 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 10 monclient(hunting): picked mon.noname-f con 0x55bb6c129a10 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55bb6bf22df0 con 0x55bb6c129a10 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55bb6befbd00 con 0x55bb6c123a10 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55bb6bf08c50 con 0x55bb6c123630 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 10 monclient(hunting): _renew_subs 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392b26ec00 10 monclient(hunting): authenticate will time out at 736.652884s 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f3929d6d640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x55bb6c123630 0x55bb6c088460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:58198/0 (socket says 172.21.15.94:58198) 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.313+0000 7f3929d6d640 1 -- 172.21.15.94:0/2428624177 learned_addr learned my addr 172.21.15.94:0/2428624177 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.312+0000 7f392956c640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x55bb6c123a10 0x55bb6c1282f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33924/0 (socket says 172.21.15.94:33924) 2024-09-16T10:29:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3929d6d640 1 -- 172.21.15.94:0/2428624177 >> v1:172.21.15.94:6793/0 conn(0x55bb6c123630 legacy=0x55bb6c088460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f392956c640 1 -- 172.21.15.94:0/2428624177 >> v1:172.21.15.94:6791/0 conn(0x55bb6c123a10 legacy=0x55bb6c1282f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023840900 0 0) 0x55bb6bf08c50 con 0x55bb6c123630 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f392afcf640 1 -- 172.21.15.94:0/2428624177 >> v1:172.21.15.152:6789/0 conn(0x55bb6c129a10 legacy=0x55bb6c12be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): my global_id is 4220 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f390c001e10 con 0x55bb6c123630 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513116151 0 0) 0x55bb6befbd00 con 0x55bb6c123a10 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): my global_id is 4198 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x55bb6bf08c50 con 0x55bb6c123a10 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31335420 0 0) 0x55bb6bf22df0 con 0x55bb6c129a10 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient(hunting): my global_id is 4177 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x55bb6befbd00 con 0x55bb6c129a10 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2934222484 0 0) 0x55bb6bf08c50 con 0x55bb6c123a10 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 >> v1:172.21.15.94:6793/0 conn(0x55bb6c123630 legacy=0x55bb6c088460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 >> v1:172.21.15.152:6789/0 conn(0x55bb6c129a10 legacy=0x55bb6c12be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: _finish_hunting 0 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 monclient: found mon.noname-c 2024-09-16T10:29:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb6befc090 con 0x55bb6c123a10 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: _finish_auth 0 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: _check_auth_tickets 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3918001a60 con 0x55bb6c123a10 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: dump: 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f392b26ec00 5 monclient: authenticate success, global_id 4198 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f392b26ec00 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3918001d70 con 0x55bb6c123a10 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f392b26ec00 10 monclient: get_monmap_and_config success 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 1 -- 172.21.15.94:0/2428624177 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3918003410 con 0x55bb6c123a10 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f392b26ec00 4 set_mon_vals no callback set 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.315+0000 7f3928d6b640 10 monclient: dump: 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.316+0000 7f392b26ec00 10 monclient: shutdown 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.316+0000 7f392b26ec00 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.316+0000 7f392b26ec00 1 -- 172.21.15.94:0/2428624177 >> v1:172.21.15.94:6791/0 conn(0x55bb6c123a10 legacy=0x55bb6c1282f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.316+0000 7f392b26ec00 1 -- 172.21.15.94:0/2428624177 shutdown_connections 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.316+0000 7f392b26ec00 1 -- 172.21.15.94:0/2428624177 >> 172.21.15.94:0/2428624177 conn(0x55bb6c090a80 msgr2=0x55bb6c072670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.316+0000 7f392b26ec00 1 -- 172.21.15.94:0/2428624177 shutdown_connections 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.316+0000 7f392b26ec00 1 -- 172.21.15.94:0/2428624177 wait complete. 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.316+0000 7f392b26ec00 10 monclient: build_initial_monmap 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.316+0000 7f392b26ec00 10 monclient: monmap: 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:29.327 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 1 Processor -- start 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 1 -- start start 2024-09-16T10:29:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 10 monclient: init 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 10 monclient: _reopen_session rank -1 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 10 monclient: _add_conns ranks=[0,3,8,7,4,5,6,1,2] 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 10 monclient(hunting): picked mon.noname-a con 0x55bb6c123630 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 10 monclient(hunting): picked mon.noname-g con 0x55bb6c123a10 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 10 monclient(hunting): picked mon.noname-e con 0x55bb6c129a10 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55bb6befbaf0 con 0x55bb6c123630 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55bb6c08c1f0 con 0x55bb6c123a10 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55bb6c08d6a0 con 0x55bb6c129a10 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 10 monclient(hunting): _renew_subs 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f392b26ec00 10 monclient(hunting): authenticate will time out at 736.657808s 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f3929d6d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x55bb6c123630 0x55bb6c08ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59336/0 (socket says 172.21.15.94:59336) 2024-09-16T10:29:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.317+0000 7f3929d6d640 1 -- 172.21.15.94:0/899049567 learned_addr learned my addr 172.21.15.94:0/899049567 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:29.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f392afcf640 1 -- 172.21.15.94:0/899049567 >> v1:172.21.15.94:6793/0 conn(0x55bb6c129a10 legacy=0x55bb6c13a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:29.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3929d6d640 1 -- 172.21.15.94:0/899049567 >> v1:172.21.15.94:6789/0 conn(0x55bb6c123630 legacy=0x55bb6c08ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:29.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176846764 0 0) 0x55bb6c08d6a0 con 0x55bb6c129a10 2024-09-16T10:29:29.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:29.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:29.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:29.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): my global_id is 4229 2024-09-16T10:29:29.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f392956c640 1 -- 172.21.15.94:0/899049567 >> v1:172.21.15.152:6790/0 conn(0x55bb6c123a10 legacy=0x55bb6c08b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3910001e10 con 0x55bb6c129a10 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564648163 0 0) 0x55bb6befbaf0 con 0x55bb6c123630 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): my global_id is 4239 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x55bb6c08d6a0 con 0x55bb6c123630 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4137315393 0 0) 0x7f3910001e10 con 0x55bb6c129a10 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55bb6befbaf0 con 0x55bb6c129a10 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f391c001ff0 con 0x55bb6c129a10 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): dump: 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:29.335 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1910238060 0 0) 0x55bb6c08d6a0 con 0x55bb6c123630 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3910001e10 con 0x55bb6c123630 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845280356 0 0) 0x55bb6c08c1f0 con 0x55bb6c123a10 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:29.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): my global_id is 4278 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x55bb6c08d6a0 con 0x55bb6c123a10 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3914001d90 con 0x55bb6c123630 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient(hunting): dump: 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:29.337 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830219166 0 0) 0x55bb6befbaf0 con 0x55bb6c129a10 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 >> v1:172.21.15.152:6790/0 conn(0x55bb6c123a10 legacy=0x55bb6c08b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 >> v1:172.21.15.94:6789/0 conn(0x55bb6c123630 legacy=0x55bb6c08ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:29.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 monclient: found mon.e 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb6c08ba80 con 0x55bb6c129a10 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient: _finish_auth 0 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f392b26ec00 5 monclient: authenticate success, global_id 4229 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f392b26ec00 10 monclient: _renew_subs 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f392b26ec00 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bb6c122d00 con 0x55bb6c129a10 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f392b26ec00 10 monclient: _renew_subs 2024-09-16T10:29:29.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f392b26ec00 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=0}) v3 -- 0x55bb6c08d3c0 con 0x55bb6c129a10 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f391c0025a0 con 0x55bb6c129a10 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f391c002a50 con 0x55bb6c129a10 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3921ffb640 10 monclient: dump: 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:29.340 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.320+0000 7f3923fff640 4 set_mon_vals no callback set 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.321+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 7 ==== mgrmap(e 4) v1 ==== 44726+0+0 (unknown 1094748937 0 0) 0x7f391c002c30 con 0x55bb6c129a10 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.321+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 8 ==== osd_map(13..13 src has 1..13) v4 ==== 3495+0+0 (unknown 813748589 0 0) 0x7f391c044980 con 0x55bb6c129a10 2024-09-16T10:29:29.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.322+0000 7f392b26ec00 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T10:29:29.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.322+0000 7f392b26ec00 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T10:29:29.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.322+0000 7f392b26ec00 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:29.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.322+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x55bb6c120390 con 0x55bb6c129a10 2024-09-16T10:29:29.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:29.322+0000 7f3929d6d640 10 monclient: get_auth_request con 0x7f3910038160 auth_method 0 2024-09-16T10:29:30.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.030+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 141+0+0 (unknown 2078359144 0 0) 0x7f391c013d50 con 0x55bb6c129a10 2024-09-16T10:29:30.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.030+0000 7f3921ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T10:29:30.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.030+0000 7f3921ffb640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T10:29:30.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.030+0000 7f392b26ec00 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:30.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.030+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x55bb6bf23990 con 0x55bb6c129a10 2024-09-16T10:29:30.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.771+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 10 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (unknown 1566986600 0 0) 0x7f391c013d50 con 0x55bb6c129a10 2024-09-16T10:29:30.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.771+0000 7f3921ffb640 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-16T10:29:30.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.771+0000 7f3923fff640 10 monclient: _renew_subs 2024-09-16T10:29:30.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.771+0000 7f3923fff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:30.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.771+0000 7f3923fff640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=14}) v3 -- 0x7f3904002bc0 con 0x55bb6c129a10 2024-09-16T10:29:30.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.771+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 11 ==== osd_map(14..14 src has 1..14) v4 ==== 1028+0+0 (unknown 215189090 0 0) 0x7f391c0141b0 con 0x55bb6c129a10 2024-09-16T10:29:30.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.771+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.152:6806/3657457384 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) v8 -- 0x55bb6c13fde0 con 0x55bb6c13c2d0 2024-09-16T10:29:30.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.771+0000 7f392956c640 10 monclient: get_auth_request con 0x55bb6c13c2d0 auth_method 0 2024-09-16T10:29:30.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.777+0000 7f392956c640 1 -- 172.21.15.94:0/899049567 <== osd.5 v1:172.21.15.152:6806/3657457384 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 ==== 153+0+0 (unknown 1742540995 0 0) 0x7f39180018a0 con 0x55bb6c13c2d0 2024-09-16T10:29:30.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.780+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.152:6806/3657457384 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x55bb6c18fa20 con 0x55bb6c13c2d0 2024-09-16T10:29:30.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.780+0000 7f392956c640 1 -- 172.21.15.94:0/899049567 <== osd.5 v1:172.21.15.152:6806/3657457384 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 173568156 0 0) 0x7f39180018a0 con 0x55bb6c13c2d0 2024-09-16T10:29:30.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.780+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.152:6806/3657457384 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x55bb6c18eb30 con 0x55bb6c13c2d0 2024-09-16T10:29:30.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.780+0000 7f392956c640 1 -- 172.21.15.94:0/899049567 <== osd.5 v1:172.21.15.152:6806/3657457384 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 185109136 0 0) 0x7f39180018a0 con 0x55bb6c13c2d0 2024-09-16T10:29:30.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.780+0000 7f39237fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:30.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:30.780+0000 7f39237fe640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7f38fc001f20 con 0x55bb6c129a10 2024-09-16T10:29:31.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.773+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 12 ==== pool_op_reply(tid 4 (0) Success v15) v1 ==== 55+0+0 (unknown 2854138364 0 0) 0x7f391c0123f0 con 0x55bb6c129a10 2024-09-16T10:29:31.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.773+0000 7f3921ffb640 10 monclient: _renew_subs 2024-09-16T10:29:31.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.773+0000 7f3921ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:31.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.773+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=15}) v3 -- 0x55bb6befbaf0 con 0x55bb6c129a10 2024-09-16T10:29:31.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.773+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 13 ==== osd_map(15..15 src has 1..15) v4 ==== 626+0+0 (unknown 3623870434 0 0) 0x7f391c01c570 con 0x55bb6c129a10 2024-09-16T10:29:31.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.773+0000 7f3923fff640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.152:6806/3657457384 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) v8 -- 0x7f39040056e0 con 0x55bb6c13c2d0 2024-09-16T10:29:31.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.781+0000 7f392956c640 1 -- 172.21.15.94:0/899049567 <== osd.5 v1:172.21.15.152:6806/3657457384 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) v8 ==== 194+0+0 (unknown 913822303 0 0) 0x7f39180018a0 con 0x55bb6c13c2d0 2024-09-16T10:29:31.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.782+0000 7f39237fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:31.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.782+0000 7f39237fe640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v15) v4 -- 0x7f38fc0033b0 con 0x55bb6c129a10 2024-09-16T10:29:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:31.856+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 14 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f391c010480 con 0x55bb6c129a10 2024-09-16T10:29:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.317+0000 7f3920ff9640 10 monclient: tick 2024-09-16T10:29:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.317+0000 7f3920ff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.317+0000 7f3920ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.318+0000 7f3920ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:32.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.784+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 15 ==== pool_op_reply(tid 6 (0) Success v16) v1 ==== 43+0+0 (unknown 957380338 0 0) 0x7f391c010910 con 0x55bb6c129a10 2024-09-16T10:29:32.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.784+0000 7f3921ffb640 10 monclient: _renew_subs 2024-09-16T10:29:32.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.784+0000 7f3921ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:32.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.784+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=16}) v3 -- 0x55bb6c08c1f0 con 0x55bb6c129a10 2024-09-16T10:29:32.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.785+0000 7f3921ffb640 1 -- 172.21.15.94:0/899049567 <== mon.8 v1:172.21.15.94:6793/0 16 ==== osd_map(16..16 src has 1..16) v4 ==== 654+0+0 (unknown 2575902701 0 0) 0x7f391c013b70 con 0x55bb6c129a10 2024-09-16T10:29:32.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.785+0000 7f3923fff640 1 -- 172.21.15.94:0/899049567 --> v1:172.21.15.152:6806/3657457384 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x7f3904005a90 con 0x55bb6c13c2d0 2024-09-16T10:29:32.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392956c640 1 -- 172.21.15.94:0/899049567 <== osd.5 v1:172.21.15.152:6806/3657457384 5 ==== osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) v8 ==== 152+0+0 (unknown 2130366450 0 0) 0x7f39180018a0 con 0x55bb6c13c2d0 2024-09-16T10:29:32.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 >> v1:172.21.15.152:6806/3657457384 conn(0x55bb6c13c2d0 legacy=0x55bb6c13e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:32.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 >> v1:172.21.15.152:6800/2686360781 conn(0x7f3910038160 legacy=0x7f391003a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:32.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392b26ec00 10 monclient: shutdown 2024-09-16T10:29:32.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392b26ec00 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:32.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 >> v1:172.21.15.94:6793/0 conn(0x55bb6c129a10 legacy=0x55bb6c13a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:32.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392afcf640 1 -- 172.21.15.94:0/899049567 reap_dead start 2024-09-16T10:29:32.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 shutdown_connections 2024-09-16T10:29:32.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 >> 172.21.15.94:0/899049567 conn(0x55bb6c090a80 msgr2=0x55bb6c11d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:32.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 shutdown_connections 2024-09-16T10:29:32.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.794+0000 7f392b26ec00 1 -- 172.21.15.94:0/899049567 wait complete. 2024-09-16T10:29:32.800 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T10:29:32.800 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T10:29:32.800 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T10:29:32.801 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T10:29:32.801 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T10:29:32.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.901+0000 7f9bff353640 10 monclient: get_monmap_and_config 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.901+0000 7f9bff353640 10 monclient: build_initial_monmap 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.901+0000 7f9bff353640 10 monclient: monmap: 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:32.902137+0000 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:32.902137+0000 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:32.903 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:32.904 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:32.904 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:32.904 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:32.904 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:32.904 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:32.904 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:32.904 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:32.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.901+0000 7f9bff353640 1 Processor -- start 2024-09-16T10:29:32.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 1 -- start start 2024-09-16T10:29:32.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 10 monclient: init 2024-09-16T10:29:32.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 10 monclient: _add_conns ranks=[3,8,5,1,7,0,2,4,6] 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 10 monclient(hunting): picked mon.noname-g con 0x7f9bf81368e0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 10 monclient(hunting): picked mon.noname-e con 0x7f9bf8136cc0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 10 monclient(hunting): picked mon.noname-h con 0x7f9bf81370a0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf8061ae0 con 0x7f9bf81368e0 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80a03e0 con 0x7f9bf81370a0 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf806c8d0 con 0x7f9bf8136cc0 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bff353640 10 monclient(hunting): authenticate will time out at 740.242607s 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bfc8ee640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f9bf8136cc0 0x7f9bf813b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:58224/0 (socket says 172.21.15.94:58224) 2024-09-16T10:29:32.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.902+0000 7f9bfc8ee640 1 -- 172.21.15.94:0/1169400698 learned_addr learned my addr 172.21.15.94:0/1169400698 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:32.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.904+0000 7f9bfc8ee640 1 -- 172.21.15.94:0/1169400698 >> v1:172.21.15.94:6793/0 conn(0x7f9bf8136cc0 legacy=0x7f9bf813b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:32.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.905+0000 7f9bfd8f0640 1 -- 172.21.15.94:0/1169400698 >> v1:172.21.15.152:6791/0 conn(0x7f9bf81370a0 legacy=0x7f9bf813ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:32.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.905+0000 7f9bfd0ef640 1 -- 172.21.15.94:0/1169400698 >> v1:172.21.15.152:6790/0 conn(0x7f9bf81368e0 legacy=0x7f9bf80a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:32.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.995+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751428805 0 0) 0x7f9bf806c8d0 con 0x7f9bf8136cc0 2024-09-16T10:29:32.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.995+0000 7f9beffff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:32.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.995+0000 7f9beffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:32.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.995+0000 7f9beffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:32.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.995+0000 7f9beffff640 10 monclient(hunting): my global_id is 4247 2024-09-16T10:29:32.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.995+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be4003520 con 0x7f9bf8136cc0 2024-09-16T10:29:33.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385079264 0 0) 0x7f9bf80a03e0 con 0x7f9bf81370a0 2024-09-16T10:29:33.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 10 monclient(hunting): my global_id is 4244 2024-09-16T10:29:33.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf806c8d0 con 0x7f9bf81370a0 2024-09-16T10:29:33.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734923985 0 0) 0x7f9bf8061ae0 con 0x7f9bf81368e0 2024-09-16T10:29:33.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 10 monclient(hunting): my global_id is 4296 2024-09-16T10:29:33.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:32.998+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf80a03e0 con 0x7f9bf81368e0 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1195100056 0 0) 0x7f9bf80a03e0 con 0x7f9bf81368e0 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 >> v1:172.21.15.94:6793/0 conn(0x7f9bf8136cc0 legacy=0x7f9bf813b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 >> v1:172.21.15.152:6791/0 conn(0x7f9bf81370a0 legacy=0x7f9bf813ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 1 monclient: found mon.noname-g 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.152:6790/0 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80a0680 con 0x7f9bf81368e0 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 10 monclient: _finish_auth 0 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:33.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9bf4001a50 con 0x7f9bf81368e0 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9beffff640 10 monclient: dump: 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.085 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.085 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.085 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.085 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.085 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.085 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.085 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9bff353640 5 monclient: authenticate success, global_id 4296 2024-09-16T10:29:33.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.081+0000 7f9bff353640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:33.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.082+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 <== mon.3 v1:172.21.15.152:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bf4001d60 con 0x7f9bf81368e0 2024-09-16T10:29:33.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.082+0000 7f9beffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:33.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.082+0000 7f9beffff640 1 -- 172.21.15.94:0/1169400698 <== mon.3 v1:172.21.15.152:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9bf4003950 con 0x7f9bf81368e0 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.082+0000 7f9bff353640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.082+0000 7f9bff353640 4 set_mon_vals no callback set 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.082+0000 7f9beffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.082+0000 7f9beffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.082+0000 7f9beffff640 10 monclient: dump: 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.088 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.083+0000 7f9bff353640 10 monclient: shutdown 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.083+0000 7f9bff353640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.083+0000 7f9bff353640 1 -- 172.21.15.94:0/1169400698 >> v1:172.21.15.152:6790/0 conn(0x7f9bf81368e0 legacy=0x7f9bf80a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.083+0000 7f9bff353640 1 -- 172.21.15.94:0/1169400698 shutdown_connections 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.083+0000 7f9bff353640 1 -- 172.21.15.94:0/1169400698 >> 172.21.15.94:0/1169400698 conn(0x7f9bf80a58d0 msgr2=0x7f9bf80a5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.083+0000 7f9bff353640 1 -- 172.21.15.94:0/1169400698 shutdown_connections 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.083+0000 7f9bff353640 1 -- 172.21.15.94:0/1169400698 wait complete. 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.083+0000 7f9bff353640 10 monclient: build_initial_monmap 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.083+0000 7f9bff353640 10 monclient: monmap: 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:33.090 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 1 Processor -- start 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 1 -- start start 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 10 monclient: init 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 10 monclient: _add_conns ranks=[8,5,0,1,3,6,4,7,2] 2024-09-16T10:29:33.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 10 monclient(hunting): picked mon.noname-e con 0x7f9bf81368e0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 10 monclient(hunting): picked mon.noname-h con 0x7f9bf8136cc0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf81370a0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80a2a40 con 0x7f9bf81370a0 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80a2580 con 0x7f9bf8136cc0 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80a87f0 con 0x7f9bf81368e0 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bff353640 10 monclient(hunting): authenticate will time out at 740.424878s 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bfd8f0640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9bf81370a0 0x7f9bf81d3410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59350/0 (socket says 172.21.15.94:59350) 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bfd8f0640 1 -- 172.21.15.94:0/1928761358 learned_addr learned my addr 172.21.15.94:0/1928761358 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:33.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.084+0000 7f9bfd0ef640 1 --1- 172.21.15.94:0/1928761358 >> v1:172.21.15.94:6793/0 conn(0x7f9bf81368e0 0x7f9bf81357b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:58236/0 (socket says 172.21.15.94:58236) 2024-09-16T10:29:33.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bfd8f0640 1 -- 172.21.15.94:0/1928761358 >> v1:172.21.15.94:6789/0 conn(0x7f9bf81370a0 legacy=0x7f9bf81d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bfd0ef640 1 -- 172.21.15.94:0/1928761358 >> v1:172.21.15.94:6793/0 conn(0x7f9bf81368e0 legacy=0x7f9bf81357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bfc8ee640 1 -- 172.21.15.94:0/1928761358 >> v1:172.21.15.152:6791/0 conn(0x7f9bf8136cc0 legacy=0x7f9bf81cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455034895 0 0) 0x7f9bf80a87f0 con 0x7f9bf81368e0 2024-09-16T10:29:33.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): my global_id is 4256 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bdc003520 con 0x7f9bf81368e0 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224342650 0 0) 0x7f9bf806c8d0 con 0x7f9bf81370a0 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): my global_id is 4257 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf80a87f0 con 0x7f9bf81370a0 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408812154 0 0) 0x7f9bf80a2580 con 0x7f9bf8136cc0 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): my global_id is 4253 2024-09-16T10:29:33.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf806c8d0 con 0x7f9bf8136cc0 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272748529 0 0) 0x7f9bf80a87f0 con 0x7f9bf81370a0 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf80a2580 con 0x7f9bf81370a0 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9bf0002020 con 0x7f9bf81370a0 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): dump: 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.098 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1560134313 0 0) 0x7f9bdc003520 con 0x7f9bf81368e0 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf80a87f0 con 0x7f9bf81368e0 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9bf4003950 con 0x7f9bf81368e0 2024-09-16T10:29:33.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): dump: 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.100 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2423554658 0 0) 0x7f9bf806c8d0 con 0x7f9bf8136cc0 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bdc003520 con 0x7f9bf8136cc0 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9be80019e0 con 0x7f9bf8136cc0 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient(hunting): dump: 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.101 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.102 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3137316340 0 0) 0x7f9bf80a2580 con 0x7f9bf81370a0 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 >> v1:172.21.15.94:6793/0 conn(0x7f9bf81368e0 legacy=0x7f9bf81357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 >> v1:172.21.15.152:6791/0 conn(0x7f9bf8136cc0 legacy=0x7f9bf81cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient: _finish_hunting 0 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 monclient: found mon.a 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf8121970 con 0x7f9bf81370a0 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient: _finish_auth 0 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient: _check_auth_tickets 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bff353640 5 monclient: authenticate success, global_id 4257 2024-09-16T10:29:33.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bff353640 10 monclient: _renew_subs 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bff353640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf8122e90 con 0x7f9bf81370a0 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bf0003d60 con 0x7f9bf81370a0 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9bf0004210 con 0x7f9bf81370a0 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bedffb640 10 monclient: dump: 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.104 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.087+0000 7f9bef7fe640 4 set_mon_vals no callback set 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.088+0000 7f9bff353640 10 monclient: _renew_subs 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.088+0000 7f9bff353640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.088+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf8120da0 con 0x7f9bf81370a0 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.088+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f9bf00043f0 con 0x7f9bf81370a0 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.089+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f9bf0046030 con 0x7f9bf81370a0 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.089+0000 7f9bff353640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.089+0000 7f9bff353640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.089+0000 7f9bff353640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.089+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf80a2a40 con 0x7f9bf81370a0 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.089+0000 7f9bfd0ef640 10 monclient: get_auth_request con 0x7f9bdc039d80 auth_method 0 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.093+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f9bf80a2a40 con 0x7f9bf81370a0 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.093+0000 7f9bedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:33.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.093+0000 7f9bedffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.298+0000 7f9bff353640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T10:29:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.298+0000 7f9bff353640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T10:29:33.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.298+0000 7f9bff353640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:33.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.298+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f9bf80a2a40 con 0x7f9bf81370a0 2024-09-16T10:29:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.299+0000 7f9bedffb640 1 -- 172.21.15.94:0/1928761358 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+86085 (unknown 481731318 0 1362842233) 0x7f9bf80a2a40 con 0x7f9bf81370a0 2024-09-16T10:29:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.299+0000 7f9bedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T10:29:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.299+0000 7f9bedffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:33.302 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:33.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.301+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 >> v1:172.21.15.152:6800/2686360781 conn(0x7f9bdc039d80 legacy=0x7f9bdc03c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.301+0000 7f9bff353640 10 monclient: shutdown 2024-09-16T10:29:33.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.301+0000 7f9bff353640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:33.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.301+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 >> v1:172.21.15.94:6789/0 conn(0x7f9bf81370a0 legacy=0x7f9bf81d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.301+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 shutdown_connections 2024-09-16T10:29:33.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.301+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 >> 172.21.15.94:0/1928761358 conn(0x7f9bf80a58d0 msgr2=0x7f9bf81be530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:33.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.301+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 shutdown_connections 2024-09-16T10:29:33.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.301+0000 7f9bff353640 1 -- 172.21.15.94:0/1928761358 wait complete. 2024-09-16T10:29:33.313 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":5,"flags":0,"active_gid":4109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6800","nonce":2686360781}]},"active_addr":"172.21.15.152:6800/2686360781","active_change":"2024-09-16T10:29:20.989035+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":597546376}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":3439971274}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":3733385918}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":956660130}]}]} 2024-09-16T10:29:33.315 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T10:29:33.315 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T10:29:33.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:29:33.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.409+0000 7fc0bcec5640 10 monclient: get_monmap_and_config 2024-09-16T10:29:33.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.409+0000 7fc0bcec5640 10 monclient: build_initial_monmap 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.409+0000 7fc0bcec5640 10 monclient: monmap: 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:33.409957+0000 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:33.409957+0000 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:33.411 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:33.412 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.409+0000 7fc0bcec5640 1 Processor -- start 2024-09-16T10:29:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.409+0000 7fc0bcec5640 1 -- start start 2024-09-16T10:29:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.409+0000 7fc0bcec5640 10 monclient: init 2024-09-16T10:29:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 10 monclient: _add_conns ranks=[3,6,2,7,8,4,0,1,5] 2024-09-16T10:29:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 10 monclient(hunting): picked mon.noname-g con 0x7fc0b80a38d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:33.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 10 monclient(hunting): picked mon.noname-d con 0x7fc0b806c670 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:33.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 10 monclient(hunting): picked mon.noname-b con 0x7fc0b80a43c0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:33.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b8061ae0 con 0x7fc0b80a43c0 2024-09-16T10:29:33.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80a04d0 con 0x7fc0b80a38d0 2024-09-16T10:29:33.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80a3250 con 0x7fc0b806c670 2024-09-16T10:29:33.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:33.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0bcec5640 10 monclient(hunting): authenticate will time out at 740.750476s 2024-09-16T10:29:33.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0b5d9b640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fc0b806c670 0x7fc0b813bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:44282/0 (socket says 172.21.15.94:44282) 2024-09-16T10:29:33.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0b6d9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fc0b80a43c0 0x7fc0b813f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53368/0 (socket says 172.21.15.94:53368) 2024-09-16T10:29:33.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.410+0000 7fc0b5d9b640 1 -- 172.21.15.94:0/1895616176 learned_addr learned my addr 172.21.15.94:0/1895616176 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:33.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b6d9d640 1 -- 172.21.15.94:0/1895616176 >> v1:172.21.15.94:6790/0 conn(0x7fc0b80a43c0 legacy=0x7fc0b813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b5d9b640 1 -- 172.21.15.94:0/1895616176 >> v1:172.21.15.94:6792/0 conn(0x7fc0b806c670 legacy=0x7fc0b813bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408114279 0 0) 0x7fc0b80a3250 con 0x7fc0b806c670 2024-09-16T10:29:33.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 10 monclient(hunting): my global_id is 4173 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a0003520 con 0x7fc0b806c670 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934625587 0 0) 0x7fc0b8061ae0 con 0x7fc0b80a43c0 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 10 monclient(hunting): my global_id is 4223 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.412+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b80a3250 con 0x7fc0b80a43c0 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b659c640 1 -- 172.21.15.94:0/1895616176 >> v1:172.21.15.152:6790/0 conn(0x7fc0b80a38d0 legacy=0x7fc0b80a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 536998999 0 0) 0x7fc0a0003520 con 0x7fc0b806c670 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 >> v1:172.21.15.152:6790/0 conn(0x7fc0b80a38d0 legacy=0x7fc0b80a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 >> v1:172.21.15.94:6790/0 conn(0x7fc0b80a43c0 legacy=0x7fc0b813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 10 monclient: _finish_hunting 0 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 1 monclient: found mon.noname-d 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80a0a00 con 0x7fc0b806c670 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 10 monclient: _finish_auth 0 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 10 monclient: _check_auth_tickets 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc0ac001a50 con 0x7fc0b806c670 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 5 monclient: authenticate success, global_id 4173 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 10 monclient: dump: 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.421 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b5d9b640 1 --1- 172.21.15.94:0/1895616176 >> v1:172.21.15.94:6792/0 conn(0x7fc0b806c670 0x7fc0b813bae0 :-1 s=OPENED pgs=56 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7fc0ac001d30 config(0 keys) v1 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0ac001d30 con 0x7fc0b806c670 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 4 set_mon_vals no callback set 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 10 monclient: shutdown 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 1 -- 172.21.15.94:0/1895616176 >> v1:172.21.15.94:6792/0 conn(0x7fc0b806c670 legacy=0x7fc0b813bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 1 -- 172.21.15.94:0/1895616176 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc0ac004be0 con 0x7fc0b806c670 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0b559a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 1 -- 172.21.15.94:0/1895616176 shutdown_connections 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 1 -- 172.21.15.94:0/1895616176 >> 172.21.15.94:0/1895616176 conn(0x7fc0b80a59d0 msgr2=0x7fc0b80a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 1 -- 172.21.15.94:0/1895616176 shutdown_connections 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.413+0000 7fc0bcec5640 1 -- 172.21.15.94:0/1895616176 wait complete. 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.414+0000 7fc0bcec5640 10 monclient: build_initial_monmap 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.414+0000 7fc0bcec5640 10 monclient: monmap: 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:33.423 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.414+0000 7fc0bcec5640 1 Processor -- start 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.414+0000 7fc0bcec5640 1 -- start start 2024-09-16T10:29:33.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.414+0000 7fc0bcec5640 10 monclient: init 2024-09-16T10:29:33.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:33.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 10 monclient: _add_conns ranks=[1,4,6,7,2,0,3,5,8] 2024-09-16T10:29:33.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 10 monclient(hunting): picked mon.noname-f con 0x7fc0b806c670 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:33.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 10 monclient(hunting): picked mon.noname-c con 0x7fc0b80a38d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:33.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 10 monclient(hunting): picked mon.noname-d con 0x7fc0b80a43c0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:33.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b8125780 con 0x7fc0b806c670 2024-09-16T10:29:33.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b809efb0 con 0x7fc0b80a38d0 2024-09-16T10:29:33.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b806bb00 con 0x7fc0b80a43c0 2024-09-16T10:29:33.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:33.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0bcec5640 10 monclient(hunting): authenticate will time out at 740.755433s 2024-09-16T10:29:33.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0b6d9d640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fc0b80a43c0 0x7fc0b81d3520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:44298/0 (socket says 172.21.15.94:44298) 2024-09-16T10:29:33.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.415+0000 7fc0b6d9d640 1 -- 172.21.15.94:0/2341867382 learned_addr learned my addr 172.21.15.94:0/2341867382 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:33.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0b6d9d640 1 -- 172.21.15.94:0/2341867382 >> v1:172.21.15.94:6792/0 conn(0x7fc0b80a43c0 legacy=0x7fc0b81d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0b5d9b640 1 -- 172.21.15.94:0/2341867382 >> v1:172.21.15.94:6791/0 conn(0x7fc0b80a38d0 legacy=0x7fc0b81cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014880641 0 0) 0x7fc0b806bb00 con 0x7fc0b80a43c0 2024-09-16T10:29:33.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 10 monclient(hunting): my global_id is 4182 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc098003520 con 0x7fc0b80a43c0 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446347041 0 0) 0x7fc0b809efb0 con 0x7fc0b80a38d0 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 10 monclient(hunting): my global_id is 4216 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b806bb00 con 0x7fc0b80a38d0 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1707409660 0 0) 0x7fc098003520 con 0x7fc0b80a43c0 2024-09-16T10:29:33.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.417+0000 7fc0b659c640 1 -- 172.21.15.94:0/2341867382 >> v1:172.21.15.152:6789/0 conn(0x7fc0b806c670 legacy=0x7fc0b81369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0b809efb0 con 0x7fc0b80a43c0 2024-09-16T10:29:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc0b0002030 con 0x7fc0b80a43c0 2024-09-16T10:29:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient(hunting): dump: 2024-09-16T10:29:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.433 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.433 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:33.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4082992600 0 0) 0x7fc0b806bb00 con 0x7fc0b80a38d0 2024-09-16T10:29:33.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc098003520 con 0x7fc0b80a38d0 2024-09-16T10:29:33.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc0ac004b20 con 0x7fc0b80a38d0 2024-09-16T10:29:33.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:33.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient(hunting): dump: 2024-09-16T10:29:33.434 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.434 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.435 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.435 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.435 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.435 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.435 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3166750266 0 0) 0x7fc0b809efb0 con 0x7fc0b80a43c0 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 >> v1:172.21.15.94:6791/0 conn(0x7fc0b80a38d0 legacy=0x7fc0b81cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 >> v1:172.21.15.152:6789/0 conn(0x7fc0b806c670 legacy=0x7fc0b81369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:33.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 monclient: found mon.d 2024-09-16T10:29:33.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:33.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:33.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b809f2a0 con 0x7fc0b80a43c0 2024-09-16T10:29:33.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0bcec5640 5 monclient: authenticate success, global_id 4182 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0bcec5640 10 monclient: _renew_subs 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0bcec5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b8127400 con 0x7fc0b80a43c0 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0bcec5640 10 monclient: _renew_subs 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0bcec5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b8137100 con 0x7fc0b80a43c0 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b0001c60 con 0x7fc0b80a43c0 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:33.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc0b0001e00 con 0x7fc0b80a43c0 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0a6ffd640 10 monclient: dump: 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.439 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.418+0000 7fc0b4d99640 4 set_mon_vals no callback set 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.419+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fc0b0010410 con 0x7fc0b80a43c0 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.419+0000 7fc0bcec5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.419+0000 7fc0bcec5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.419+0000 7fc0bcec5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.419+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b804e210 con 0x7fc0b80a43c0 2024-09-16T10:29:33.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.419+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7fc0b00462a0 con 0x7fc0b80a43c0 2024-09-16T10:29:33.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.419+0000 7fc0b659c640 10 monclient: get_auth_request con 0x7fc0980397c0 auth_method 0 2024-09-16T10:29:33.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.423+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fc0b0015780 con 0x7fc0b80a43c0 2024-09-16T10:29:33.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.423+0000 7fc0a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:33.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.423+0000 7fc0a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:33.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.584+0000 7fc0bcec5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:33.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.584+0000 7fc0bcec5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:33.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.584+0000 7fc0bcec5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:33.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.584+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0b80a0fc0 con 0x7fc0b80a43c0 2024-09-16T10:29:33.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.585+0000 7fc0a6ffd640 1 -- 172.21.15.94:0/2341867382 <== mon.6 v1:172.21.15.94:6792/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10830 (unknown 1054498076 0 24213607) 0x7fc0b0013bf0 con 0x7fc0b80a43c0 2024-09-16T10:29:33.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.585+0000 7fc0a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:33.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.585+0000 7fc0a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:33.587 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:33.588 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":16,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","created":"2024-09-16T10:29:16.029056+0000","modified":"2024-09-16T10:29:32.780214+0000","last_up_change":"2024-09-16T10:29:27.994140+0000","last_in_change":"2024-09-16T10:29:20.181320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:29:27.046989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T10:29:28.942998+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"19cab4f7-c271-4ce8-895e-75a5871502f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":4202994153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":4202994153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":4202994153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":4202994153}]},"public_addr":"172.21.15.94:6804/4202994153","cluster_addr":"172.21.15.94:6806/4202994153","heartbeat_back_addr":"172.21.15.94:6810/4202994153","heartbeat_front_addr":"172.21.15.94:6808/4202994153","state":["exists","up"]},{"osd":1,"uuid":"6641fac5-e871-4401-bf30-598b633e70b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":782646670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":782646670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":782646670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":782646670}]},"public_addr":"172.21.15.94:6800/782646670","cluster_addr":"172.21.15.94:6801/782646670","heartbeat_back_addr":"172.21.15.94:6803/782646670","heartbeat_front_addr":"172.21.15.94:6802/782646670","state":["exists","up"]},{"osd":2,"uuid":"6f4cedf2-071d-44f9-87a3-a766668fcd03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3102113414}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3102113414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3102113414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3102113414}]},"public_addr":"172.21.15.94:6805/3102113414","cluster_addr":"172.21.15.94:6807/3102113414","heartbeat_back_addr":"172.21.15.94:6811/3102113414","heartbeat_front_addr":"172.21.15.94:6809/3102113414","state":["exists","up"]},{"osd":3,"uuid":"3634242e-61e0-4221-8c38-7dce4d5053ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6805","nonce":823649059}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6807","nonce":823649059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6811","nonce":823649059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6809","nonce":823649059}]},"public_addr":"172.21.15.152:6805/823649059","cluster_addr":"172.21.15.152:6807/823649059","heartbeat_back_addr":"172.21.15.152:6811/823649059","heartbeat_front_addr":"172.21.15.152:6809/823649059","state":["exists","up"]},{"osd":4,"uuid":"eed750dc-2d29-443f-a91c-d9c5abdcf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6801","nonce":1475699231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6802","nonce":1475699231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6804","nonce":1475699231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6803","nonce":1475699231}]},"public_addr":"172.21.15.152:6801/1475699231","cluster_addr":"172.21.15.152:6802/1475699231","heartbeat_back_addr":"172.21.15.152:6804/1475699231","heartbeat_front_addr":"172.21.15.152:6803/1475699231","state":["exists","up"]},{"osd":5,"uuid":"5d6daac7-1ac7-4f0a-b315-27d1333fc3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6806","nonce":3657457384}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6808","nonce":3657457384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6812","nonce":3657457384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6810","nonce":3657457384}]},"public_addr":"172.21.15.152:6806/3657457384","cluster_addr":"172.21.15.152:6808/3657457384","heartbeat_back_addr":"172.21.15.152:6812/3657457384","heartbeat_front_addr":"172.21.15.152:6810/3657457384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.818679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.874108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.957469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.872051+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.780789+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.901652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:29:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.587+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 >> v1:172.21.15.152:6800/2686360781 conn(0x7fc0980397c0 legacy=0x7fc09803bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.587+0000 7fc0bcec5640 10 monclient: shutdown 2024-09-16T10:29:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.587+0000 7fc0bcec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.587+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 >> v1:172.21.15.94:6792/0 conn(0x7fc0b80a43c0 legacy=0x7fc0b81d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.587+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 shutdown_connections 2024-09-16T10:29:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.587+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 >> 172.21.15.94:0/2341867382 conn(0x7fc0b80a59d0 msgr2=0x7fc0b809ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.587+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 shutdown_connections 2024-09-16T10:29:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.587+0000 7fc0bcec5640 1 -- 172.21.15.94:0/2341867382 wait complete. 2024-09-16T10:29:33.600 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T10:29:33.600 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:29:33.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.694+0000 7f4af60a3640 10 monclient: get_monmap_and_config 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.694+0000 7f4af60a3640 10 monclient: build_initial_monmap 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.694+0000 7f4af60a3640 10 monclient: monmap: 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:33.695080+0000 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:33.695080+0000 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:33.697 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:33.697 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:33.697 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:33.697 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:33.697 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:33.697 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:33.697 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 1 Processor -- start 2024-09-16T10:29:33.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 1 -- start start 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 10 monclient: init 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 10 monclient: _add_conns ranks=[2,1,8,4,6,7,3,0,5] 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 10 monclient(hunting): picked mon.noname-b con 0x7f4af009bb70 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 10 monclient(hunting): picked mon.noname-f con 0x7f4af009ede0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 10 monclient(hunting): picked mon.noname-e con 0x7f4af00a0aa0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af0061ae0 con 0x7f4af009ede0 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af009dbe0 con 0x7f4af009bb70 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af009d900 con 0x7f4af00a0aa0 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4af60a3640 10 monclient(hunting): authenticate will time out at 741.035766s 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4aeffff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f4af00a0aa0 0x7f4af013bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:58248/0 (socket says 172.21.15.94:58248) 2024-09-16T10:29:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4aeffff640 1 -- 172.21.15.94:0/3232289842 learned_addr learned my addr 172.21.15.94:0/3232289842 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:33.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.695+0000 7f4aef7fe640 1 --1- 172.21.15.94:0/3232289842 >> v1:172.21.15.94:6790/0 conn(0x7f4af009bb70 0x7f4af009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53380/0 (socket says 172.21.15.94:53380) 2024-09-16T10:29:33.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.697+0000 7f4aeffff640 1 -- 172.21.15.94:0/3232289842 >> v1:172.21.15.94:6793/0 conn(0x7f4af00a0aa0 legacy=0x7f4af013bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aef7fe640 1 -- 172.21.15.94:0/3232289842 >> v1:172.21.15.94:6790/0 conn(0x7f4af009bb70 legacy=0x7f4af009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505385369 0 0) 0x7f4af009d900 con 0x7f4af00a0aa0 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient(hunting): my global_id is 4265 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4adc003520 con 0x7f4af00a0aa0 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838244660 0 0) 0x7f4af009dbe0 con 0x7f4af009bb70 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient(hunting): my global_id is 4232 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af009d900 con 0x7f4af009bb70 2024-09-16T10:29:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4ae7fff640 1 -- 172.21.15.94:0/3232289842 >> v1:172.21.15.152:6789/0 conn(0x7f4af009ede0 legacy=0x7f4af0138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745690415 0 0) 0x7f4adc003520 con 0x7f4af00a0aa0 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 >> v1:172.21.15.94:6790/0 conn(0x7f4af009bb70 legacy=0x7f4af009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 >> v1:172.21.15.152:6789/0 conn(0x7f4af009ede0 legacy=0x7f4af0138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 monclient: found mon.noname-e 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af009d620 con 0x7f4af00a0aa0 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4af60a3640 5 monclient: authenticate success, global_id 4265 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4ae8001e20 con 0x7f4af00a0aa0 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4af60a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: dump: 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.706 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4af60a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ae8004c20 con 0x7f4af00a0aa0 2024-09-16T10:29:33.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4af60a3640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 1 -- 172.21.15.94:0/3232289842 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4ae80050d0 con 0x7f4af00a0aa0 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4af60a3640 4 set_mon_vals no callback set 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.698+0000 7f4aeeffd640 10 monclient: dump: 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 10 monclient: shutdown 2024-09-16T10:29:33.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 1 -- 172.21.15.94:0/3232289842 >> v1:172.21.15.94:6793/0 conn(0x7f4af00a0aa0 legacy=0x7f4af013bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 1 -- 172.21.15.94:0/3232289842 shutdown_connections 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 1 -- 172.21.15.94:0/3232289842 >> 172.21.15.94:0/3232289842 conn(0x7f4af00a1680 msgr2=0x7f4af00a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 1 -- 172.21.15.94:0/3232289842 shutdown_connections 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 1 -- 172.21.15.94:0/3232289842 wait complete. 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 10 monclient: build_initial_monmap 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 10 monclient: monmap: 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:33.710 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.699+0000 7f4af60a3640 1 Processor -- start 2024-09-16T10:29:33.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 1 -- start start 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 10 monclient: init 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 10 monclient: _add_conns ranks=[4,5,0,8,3,1,6,7,2] 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 10 monclient(hunting): picked mon.noname-c con 0x7f4af009bb70 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 10 monclient(hunting): picked mon.noname-h con 0x7f4af009ede0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 10 monclient(hunting): picked mon.noname-a con 0x7f4af00a0aa0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af0070aa0 con 0x7f4af00a0aa0 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af009b3b0 con 0x7f4af009bb70 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af01210b0 con 0x7f4af009ede0 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4af60a3640 10 monclient(hunting): authenticate will time out at 741.040523s 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4aeffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4af00a0aa0 0x7f4af01cf060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59356/0 (socket says 172.21.15.94:59356) 2024-09-16T10:29:33.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4aef7fe640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f4af009bb70 0x7f4af01336d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33938/0 (socket says 172.21.15.94:33938) 2024-09-16T10:29:33.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.700+0000 7f4aeffff640 1 -- 172.21.15.94:0/3156203445 learned_addr learned my addr 172.21.15.94:0/3156203445 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:33.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aef7fe640 1 -- 172.21.15.94:0/3156203445 >> v1:172.21.15.94:6791/0 conn(0x7f4af009bb70 legacy=0x7f4af01336d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aeffff640 1 -- 172.21.15.94:0/3156203445 >> v1:172.21.15.94:6789/0 conn(0x7f4af00a0aa0 legacy=0x7f4af01cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44088343 0 0) 0x7f4af009b3b0 con 0x7f4af009bb70 2024-09-16T10:29:33.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 10 monclient(hunting): my global_id is 4225 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad0003520 con 0x7f4af009bb70 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120945651 0 0) 0x7f4af0070aa0 con 0x7f4af00a0aa0 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 10 monclient(hunting): my global_id is 4266 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af009b3b0 con 0x7f4af00a0aa0 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.702+0000 7f4ae7fff640 1 -- 172.21.15.94:0/3156203445 >> v1:172.21.15.152:6791/0 conn(0x7f4af009ede0 legacy=0x7f4af01cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525073875 0 0) 0x7f4ad0003520 con 0x7f4af009bb70 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af0070aa0 con 0x7f4af009bb70 2024-09-16T10:29:33.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4ae0001d20 con 0x7f4af009bb70 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): dump: 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.718 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2269575656 0 0) 0x7f4af009b3b0 con 0x7f4af00a0aa0 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ad0003520 con 0x7f4af00a0aa0 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4ae8002030 con 0x7f4af00a0aa0 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:33.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): dump: 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796484839 0 0) 0x7f4af01210b0 con 0x7f4af009ede0 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient(hunting): my global_id is 4262 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af009b3b0 con 0x7f4af009ede0 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 328816295 0 0) 0x7f4af0070aa0 con 0x7f4af009bb70 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 >> v1:172.21.15.152:6791/0 conn(0x7f4af009ede0 legacy=0x7f4af01cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 >> v1:172.21.15.94:6789/0 conn(0x7f4af00a0aa0 legacy=0x7f4af01cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient: _finish_hunting 0 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 monclient: found mon.c 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af0070550 con 0x7f4af009bb70 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient: _finish_auth 0 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4af60a3640 5 monclient: authenticate success, global_id 4225 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4af60a3640 10 monclient: _renew_subs 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4af60a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:33.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0120dd0 con 0x7f4af009bb70 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4af60a3640 10 monclient: _renew_subs 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4af60a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af0133de0 con 0x7f4af009bb70 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ae0004c10 con 0x7f4af009bb70 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4ae00050c0 con 0x7f4af009bb70 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aecff9640 10 monclient: dump: 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:33.723 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:33.724 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.703+0000 7f4aee7fc640 4 set_mon_vals no callback set 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.704+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f4ae00101a0 con 0x7f4af009bb70 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.704+0000 7f4af60a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.704+0000 7f4af60a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.704+0000 7f4af60a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.704+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af00a1ff0 con 0x7f4af009bb70 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.704+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f4ae0046000 con 0x7f4af009bb70 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.704+0000 7f4ae7fff640 10 monclient: get_auth_request con 0x7f4ad00398d0 auth_method 0 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.708+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f4ae00154e0 con 0x7f4af009bb70 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.708+0000 7f4aecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:33.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.708+0000 7f4aecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:33.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.869+0000 7f4af60a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:33.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.869+0000 7f4af60a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:33.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.869+0000 7f4af60a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:33.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.869+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4af009c630 con 0x7f4af009bb70 2024-09-16T10:29:33.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.870+0000 7f4aecff9640 1 -- 172.21.15.94:0/3156203445 <== mon.4 v1:172.21.15.94:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10830 (unknown 1054498076 0 24213607) 0x7f4ae0013950 con 0x7f4af009bb70 2024-09-16T10:29:33.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.870+0000 7f4aecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:33.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.870+0000 7f4aecff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:33.872 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:33.873 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":16,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","created":"2024-09-16T10:29:16.029056+0000","modified":"2024-09-16T10:29:32.780214+0000","last_up_change":"2024-09-16T10:29:27.994140+0000","last_in_change":"2024-09-16T10:29:20.181320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:29:27.046989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T10:29:28.942998+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"19cab4f7-c271-4ce8-895e-75a5871502f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":4202994153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":4202994153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":4202994153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":4202994153}]},"public_addr":"172.21.15.94:6804/4202994153","cluster_addr":"172.21.15.94:6806/4202994153","heartbeat_back_addr":"172.21.15.94:6810/4202994153","heartbeat_front_addr":"172.21.15.94:6808/4202994153","state":["exists","up"]},{"osd":1,"uuid":"6641fac5-e871-4401-bf30-598b633e70b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":782646670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":782646670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":782646670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":782646670}]},"public_addr":"172.21.15.94:6800/782646670","cluster_addr":"172.21.15.94:6801/782646670","heartbeat_back_addr":"172.21.15.94:6803/782646670","heartbeat_front_addr":"172.21.15.94:6802/782646670","state":["exists","up"]},{"osd":2,"uuid":"6f4cedf2-071d-44f9-87a3-a766668fcd03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3102113414}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3102113414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3102113414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3102113414}]},"public_addr":"172.21.15.94:6805/3102113414","cluster_addr":"172.21.15.94:6807/3102113414","heartbeat_back_addr":"172.21.15.94:6811/3102113414","heartbeat_front_addr":"172.21.15.94:6809/3102113414","state":["exists","up"]},{"osd":3,"uuid":"3634242e-61e0-4221-8c38-7dce4d5053ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6805","nonce":823649059}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6807","nonce":823649059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6811","nonce":823649059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6809","nonce":823649059}]},"public_addr":"172.21.15.152:6805/823649059","cluster_addr":"172.21.15.152:6807/823649059","heartbeat_back_addr":"172.21.15.152:6811/823649059","heartbeat_front_addr":"172.21.15.152:6809/823649059","state":["exists","up"]},{"osd":4,"uuid":"eed750dc-2d29-443f-a91c-d9c5abdcf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6801","nonce":1475699231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6802","nonce":1475699231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6804","nonce":1475699231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6803","nonce":1475699231}]},"public_addr":"172.21.15.152:6801/1475699231","cluster_addr":"172.21.15.152:6802/1475699231","heartbeat_back_addr":"172.21.15.152:6804/1475699231","heartbeat_front_addr":"172.21.15.152:6803/1475699231","state":["exists","up"]},{"osd":5,"uuid":"5d6daac7-1ac7-4f0a-b315-27d1333fc3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6806","nonce":3657457384}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6808","nonce":3657457384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6812","nonce":3657457384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6810","nonce":3657457384}]},"public_addr":"172.21.15.152:6806/3657457384","cluster_addr":"172.21.15.152:6808/3657457384","heartbeat_back_addr":"172.21.15.152:6812/3657457384","heartbeat_front_addr":"172.21.15.152:6810/3657457384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.818679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.874108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.957469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.872051+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.780789+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.901652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:29:33.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.872+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 >> v1:172.21.15.152:6800/2686360781 conn(0x7f4ad00398d0 legacy=0x7f4ad003bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.872+0000 7f4af60a3640 10 monclient: shutdown 2024-09-16T10:29:33.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.872+0000 7f4af60a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:33.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.872+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 >> v1:172.21.15.94:6791/0 conn(0x7f4af009bb70 legacy=0x7f4af01336d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:33.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.872+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 shutdown_connections 2024-09-16T10:29:33.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.872+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 >> 172.21.15.94:0/3156203445 conn(0x7f4af00a1680 msgr2=0x7f4af006c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:33.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.872+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 shutdown_connections 2024-09-16T10:29:33.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.872+0000 7f4af60a3640 1 -- 172.21.15.94:0/3156203445 wait complete. 2024-09-16T10:29:33.884 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T10:29:33.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T10:29:33.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T10:29:33.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T10:29:33.886 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T10:29:33.886 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T10:29:34.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.999+0000 7f905423a640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.999+0000 7f905423a640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.999+0000 7f905423a640 10 monclient: monmap: 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:33.999906+0000 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:33.999906+0000 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.002 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.999+0000 7effdcbe9640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.999+0000 7effdcbe9640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.999+0000 7effdcbe9640 10 monclient: monmap: 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.000149+0000 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.000149+0000 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.004 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:33.999+0000 7f905423a640 1 Processor -- start 2024-09-16T10:29:34.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.000+0000 7effdcbe9640 1 Processor -- start 2024-09-16T10:29:34.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.000+0000 7effdcbe9640 1 -- start start 2024-09-16T10:29:34.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.000+0000 7effdcbe9640 10 monclient: init 2024-09-16T10:29:34.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.000+0000 7f905423a640 1 -- start start 2024-09-16T10:29:34.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.000+0000 7f905423a640 10 monclient: init 2024-09-16T10:29:34.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 10 monclient: _add_conns ranks=[3,6,7,2,4,0,5,1,8] 2024-09-16T10:29:34.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 10 monclient: _add_conns ranks=[2,5,7,4,3,1,0,8,6] 2024-09-16T10:29:34.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 10 monclient(hunting): picked mon.noname-b con 0x7f904c0a25b0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 10 monclient(hunting): picked mon.noname-g con 0x7effd80a25b0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:34.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 10 monclient(hunting): picked mon.noname-d con 0x7effd80a30a0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 10 monclient(hunting): picked mon.noname-i con 0x7effd8071020 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd8061ae0 con 0x7effd80a25b0 2024-09-16T10:29:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effd5d9b640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7effd80a30a0 0x7effd81395a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:44306/0 (socket says 172.21.15.94:44306) 2024-09-16T10:29:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effd5d9b640 1 -- 172.21.15.94:0/1496061822 learned_addr learned my addr 172.21.15.94:0/1496061822 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 1 -- 172.21.15.94:0/1496061822 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd8070110 con 0x7effd80a30a0 2024-09-16T10:29:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f9051fd6640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f904c0a25b0 0x7f904c0a2990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53390/0 (socket says 172.21.15.94:53390) 2024-09-16T10:29:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f9051fd6640 1 -- 172.21.15.94:0/107623702 learned_addr learned my addr 172.21.15.94:0/107623702 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 1 -- 172.21.15.94:0/1496061822 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd8071890 con 0x7effd8071020 2024-09-16T10:29:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 10 monclient(hunting): picked mon.noname-h con 0x7f904c0a30a0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 10 monclient(hunting): picked mon.noname-i con 0x7f904c071020 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7effdcbe9640 10 monclient(hunting): authenticate will time out at 741.341747s 2024-09-16T10:29:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 1 -- 172.21.15.94:0/107623702 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c061ae0 con 0x7f904c0a25b0 2024-09-16T10:29:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 1 -- 172.21.15.94:0/107623702 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c070110 con 0x7f904c0a30a0 2024-09-16T10:29:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 1 -- 172.21.15.94:0/107623702 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c071890 con 0x7f904c071020 2024-09-16T10:29:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.001+0000 7f905423a640 10 monclient(hunting): authenticate will time out at 741.341940s 2024-09-16T10:29:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7f9051fd6640 1 -- 172.21.15.94:0/107623702 >> v1:172.21.15.94:6790/0 conn(0x7f904c0a25b0 legacy=0x7f904c0a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7effd659c640 1 -- 172.21.15.94:0/1496061822 >> v1:172.21.15.152:6790/0 conn(0x7effd80a25b0 legacy=0x7effd80a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7effd5d9b640 1 -- 172.21.15.94:0/1496061822 >> v1:172.21.15.94:6792/0 conn(0x7effd80a30a0 legacy=0x7effd81395a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7f9050fd4640 1 -- 172.21.15.94:0/107623702 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251587065 0 0) 0x7f904c061ae0 con 0x7f904c0a25b0 2024-09-16T10:29:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7f9050fd4640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7f9050fd4640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7f9050fd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7f9050fd4640 10 monclient(hunting): my global_id is 4241 2024-09-16T10:29:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7f9050fd4640 1 -- 172.21.15.94:0/107623702 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9038003520 con 0x7f904c0a25b0 2024-09-16T10:29:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.003+0000 7effd6d9d640 1 -- 172.21.15.94:0/1496061822 >> v1:172.21.15.152:6792/0 conn(0x7effd8071020 legacy=0x7effd813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822567413 0 0) 0x7effd8061ae0 con 0x7effd80a25b0 2024-09-16T10:29:34.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f90517d5640 1 -- 172.21.15.94:0/107623702 >> v1:172.21.15.152:6791/0 conn(0x7f904c0a30a0 legacy=0x7f904c1395a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 1 -- 172.21.15.94:0/107623702 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2224435687 0 0) 0x7f9038003520 con 0x7f904c0a25b0 2024-09-16T10:29:34.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 1 -- 172.21.15.94:0/107623702 >> v1:172.21.15.152:6792/0 conn(0x7f904c071020 legacy=0x7f904c13ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:29:34.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 1 -- 172.21.15.94:0/107623702 >> v1:172.21.15.152:6791/0 conn(0x7f904c0a30a0 legacy=0x7f904c1395a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 1 monclient: found mon.noname-b 2024-09-16T10:29:34.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 1 -- 172.21.15.94:0/107623702 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c06fda0 con 0x7f904c0a25b0 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 1 -- 172.21.15.94:0/107623702 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9048001a60 con 0x7f904c0a25b0 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: dump: 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.015 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): my global_id is 4314 2024-09-16T10:29:34.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc0003520 con 0x7effd80a25b0 2024-09-16T10:29:34.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954058440 0 0) 0x7effd8071890 con 0x7effd8071020 2024-09-16T10:29:34.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): my global_id is 4273 2024-09-16T10:29:34.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd8061ae0 con 0x7effd8071020 2024-09-16T10:29:34.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995644766 0 0) 0x7effd8070110 con 0x7effd80a30a0 2024-09-16T10:29:34.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 10 monclient(hunting): my global_id is 4191 2024-09-16T10:29:34.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd8071890 con 0x7effd80a30a0 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 1 -- 172.21.15.94:0/107623702 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9048001d70 con 0x7f904c0a25b0 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 1 -- 172.21.15.94:0/107623702 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9048004be0 con 0x7f904c0a25b0 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f905423a640 5 monclient: authenticate success, global_id 4241 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f9050fd4640 10 monclient: dump: 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.018 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f905423a640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:34.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7f905423a640 4 set_mon_vals no callback set 2024-09-16T10:29:34.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.004+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2592158161 0 0) 0x7effc0003520 con 0x7effd80a25b0 2024-09-16T10:29:34.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 >> v1:172.21.15.152:6792/0 conn(0x7effd8071020 legacy=0x7effd813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 >> v1:172.21.15.94:6792/0 conn(0x7effd80a30a0 legacy=0x7effd81395a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 1 monclient: found mon.noname-g 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.152:6790/0 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd806fda0 con 0x7effd80a25b0 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7effc8001ab0 con 0x7effd80a25b0 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:29:34.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: dump: 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.022 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.023 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.023 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.023 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effdcbe9640 5 monclient: authenticate success, global_id 4314 2024-09-16T10:29:34.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effdcbe9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 <== mon.3 v1:172.21.15.152:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effc8003610 con 0x7effd80a25b0 2024-09-16T10:29:34.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7f905423a640 10 monclient: shutdown 2024-09-16T10:29:34.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7f905423a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7f905423a640 1 -- 172.21.15.94:0/107623702 >> v1:172.21.15.94:6790/0 conn(0x7f904c0a25b0 legacy=0x7f904c0a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 1 -- 172.21.15.94:0/1496061822 <== mon.3 v1:172.21.15.152:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7effc8003ac0 con 0x7effd80a25b0 2024-09-16T10:29:34.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7f905423a640 1 -- 172.21.15.94:0/107623702 shutdown_connections 2024-09-16T10:29:34.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7f905423a640 1 -- 172.21.15.94:0/107623702 >> 172.21.15.94:0/107623702 conn(0x7f904c0a7290 msgr2=0x7f904c0a36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effdcbe9640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:34.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effdcbe9640 4 set_mon_vals no callback set 2024-09-16T10:29:34.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:34.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7f905423a640 1 -- 172.21.15.94:0/107623702 shutdown_connections 2024-09-16T10:29:34.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7f905423a640 1 -- 172.21.15.94:0/107623702 wait complete. 2024-09-16T10:29:34.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.005+0000 7effd559a640 10 monclient: dump: 2024-09-16T10:29:34.026 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.026 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.027 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.028 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.006+0000 7effdcbe9640 10 monclient: shutdown 2024-09-16T10:29:34.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.006+0000 7effdcbe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.006+0000 7effdcbe9640 1 -- 172.21.15.94:0/1496061822 >> v1:172.21.15.152:6790/0 conn(0x7effd80a25b0 legacy=0x7effd80a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.006+0000 7effdcbe9640 1 -- 172.21.15.94:0/1496061822 shutdown_connections 2024-09-16T10:29:34.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.006+0000 7effdcbe9640 1 -- 172.21.15.94:0/1496061822 >> 172.21.15.94:0/1496061822 conn(0x7effd80a7290 msgr2=0x7effd80a36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.006+0000 7effdcbe9640 1 -- 172.21.15.94:0/1496061822 shutdown_connections 2024-09-16T10:29:34.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.006+0000 7f905423a640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.006+0000 7f905423a640 10 monclient: monmap: 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.029 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.030 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.030 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.030 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.030 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.006+0000 7effdcbe9640 1 -- 172.21.15.94:0/1496061822 wait complete. 2024-09-16T10:29:34.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f905423a640 1 Processor -- start 2024-09-16T10:29:34.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f905423a640 1 -- start start 2024-09-16T10:29:34.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f905423a640 10 monclient: init 2024-09-16T10:29:34.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f905423a640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f905423a640 10 monclient: _add_conns ranks=[7,6,5,2,1,8,4,0,3] 2024-09-16T10:29:34.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f905423a640 10 monclient(hunting): picked mon.noname-i con 0x7f904c071020 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f905423a640 10 monclient(hunting): picked mon.noname-d con 0x7f904c0a25b0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7effdcbe9640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f9051fd6640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7f904c071020 0x7f904c134880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.94:35768/0 (socket says 172.21.15.94:35768) 2024-09-16T10:29:34.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f905423a640 10 monclient(hunting): picked mon.noname-h con 0x7f904c0a30a0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:34.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7effdcbe9640 10 monclient: monmap: 2024-09-16T10:29:34.032 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.032 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:34.032 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:34.032 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:34.032 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.033 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f9051fd6640 1 -- 172.21.15.94:0/2761075692 learned_addr learned my addr 172.21.15.94:0/2761075692 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.007+0000 7f905423a640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c124f90 con 0x7f904c0a30a0 2024-09-16T10:29:34.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.008+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c09fec0 con 0x7f904c0a25b0 2024-09-16T10:29:34.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.008+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c0a03a0 con 0x7f904c071020 2024-09-16T10:29:34.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.008+0000 7f905423a640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.008+0000 7f905423a640 10 monclient(hunting): authenticate will time out at 741.348192s 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.008+0000 7effdcbe9640 1 Processor -- start 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.008+0000 7effdcbe9640 1 -- start start 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.008+0000 7effdcbe9640 10 monclient: init 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.009+0000 7effdcbe9640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.009+0000 7effdcbe9640 10 monclient: _add_conns ranks=[4,2,1,8,0,3,5,7,6] 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.009+0000 7effdcbe9640 10 monclient(hunting): picked mon.noname-c con 0x7effd8071020 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.009+0000 7effd659c640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7effd8071020 0x7effd8134750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33940/0 (socket says 172.21.15.94:33940) 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.009+0000 7effd659c640 1 -- 172.21.15.94:0/95169698 learned_addr learned my addr 172.21.15.94:0/95169698 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.009+0000 7effdcbe9640 10 monclient(hunting): picked mon.noname-b con 0x7effd80a25b0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7effdcbe9640 10 monclient(hunting): picked mon.noname-f con 0x7effd80a30a0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:34.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f9051fd6640 1 -- 172.21.15.94:0/2761075692 >> v1:172.21.15.152:6792/0 conn(0x7f904c071020 legacy=0x7f904c134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd8123120 con 0x7effd80a30a0 2024-09-16T10:29:34.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd806cad0 con 0x7effd80a25b0 2024-09-16T10:29:34.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd80a6890 con 0x7effd8071020 2024-09-16T10:29:34.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90517d5640 1 -- 172.21.15.94:0/2761075692 >> v1:172.21.15.94:6792/0 conn(0x7f904c0a25b0 legacy=0x7f904c1cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90527d7640 1 -- 172.21.15.94:0/2761075692 >> v1:172.21.15.152:6791/0 conn(0x7f904c0a30a0 legacy=0x7f904c1d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7effdcbe9640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7effdcbe9640 10 monclient(hunting): authenticate will time out at 741.350359s 2024-09-16T10:29:34.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493078255 0 0) 0x7f904c0a03a0 con 0x7f904c071020 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): my global_id is 4282 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9030003520 con 0x7f904c071020 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074929565 0 0) 0x7f904c09fec0 con 0x7f904c0a25b0 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): my global_id is 4200 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c0a03a0 con 0x7f904c0a25b0 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293311737 0 0) 0x7f904c124f90 con 0x7f904c0a30a0 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 10 monclient(hunting): my global_id is 4280 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.010+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c09fec0 con 0x7f904c0a30a0 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1508537576 0 0) 0x7f9030003520 con 0x7f904c071020 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f904c124f90 con 0x7f904c071020 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9048004b20 con 0x7f904c071020 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient(hunting): dump: 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.044 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4232706522 0 0) 0x7f904c09fec0 con 0x7f904c0a30a0 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9030003520 con 0x7f904c0a30a0 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9044004600 con 0x7f904c0a30a0 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:34.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient(hunting): dump: 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2479068533 0 0) 0x7f904c124f90 con 0x7f904c071020 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 >> v1:172.21.15.94:6792/0 conn(0x7f904c0a25b0 legacy=0x7f904c1cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 >> v1:172.21.15.152:6791/0 conn(0x7f904c0a30a0 legacy=0x7f904c1d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 monclient: found mon.i 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c0a62a0 con 0x7f904c071020 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f905423a640 5 monclient: authenticate success, global_id 4282 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f905423a640 10 monclient: _renew_subs 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f905423a640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c123420 con 0x7f904c071020 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f905423a640 10 monclient: _renew_subs 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9048001ac0 con 0x7f904c071020 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f905423a640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c122f10 con 0x7f904c071020 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f90480054d0 con 0x7f904c071020 2024-09-16T10:29:34.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f90427fc640 10 monclient: dump: 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.049 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.050 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.050 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.050 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.050 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.050 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7f9043fff640 4 set_mon_vals no callback set 2024-09-16T10:29:34.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.011+0000 7effd659c640 1 -- 172.21.15.94:0/95169698 >> v1:172.21.15.94:6791/0 conn(0x7effd8071020 legacy=0x7effd8134750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7effd5d9b640 1 -- 172.21.15.94:0/95169698 >> v1:172.21.15.94:6790/0 conn(0x7effd80a25b0 legacy=0x7effd81cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056432950 0 0) 0x7effd806cad0 con 0x7effd80a25b0 2024-09-16T10:29:34.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7effc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f90480105b0 con 0x7f904c071020 2024-09-16T10:29:34.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7f905423a640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.152:6792/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f904c0a00f0 con 0x7f904c071020 2024-09-16T10:29:34.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7effc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7effc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7effc6ffd640 10 monclient(hunting): my global_id is 4250 2024-09-16T10:29:34.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f9048046250 con 0x7f904c071020 2024-09-16T10:29:34.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.94:6800/782646670 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9030044260 con 0x7f90300405b0 2024-09-16T10:29:34.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (unknown 3847591702 0 0) 0x7f9048015500 con 0x7f904c071020 2024-09-16T10:29:34.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7f90427fc640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T10:29:34.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7f90517d5640 10 monclient: get_auth_request con 0x7f90300399e0 auth_method 0 2024-09-16T10:29:34.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7f90527d7640 10 monclient: get_auth_request con 0x7f90300405b0 auth_method 0 2024-09-16T10:29:34.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb8003550 con 0x7effd80a25b0 2024-09-16T10:29:34.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.012+0000 7effd6d9d640 1 -- 172.21.15.94:0/95169698 >> v1:172.21.15.152:6789/0 conn(0x7effd80a30a0 legacy=0x7effd81cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936226581 0 0) 0x7effd8123120 con 0x7effd80a30a0 2024-09-16T10:29:34.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): my global_id is 4213 2024-09-16T10:29:34.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd806cad0 con 0x7effd80a30a0 2024-09-16T10:29:34.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2095869596 0 0) 0x7effb8003550 con 0x7effd80a25b0 2024-09-16T10:29:34.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effd8123120 con 0x7effd80a25b0 2024-09-16T10:29:34.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7effcc001bb0 con 0x7effd80a25b0 2024-09-16T10:29:34.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:34.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): dump: 2024-09-16T10:29:34.058 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.058 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106844799 0 0) 0x7effd80a6890 con 0x7effd8071020 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): my global_id is 4234 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb8003550 con 0x7effd8071020 2024-09-16T10:29:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 123143478 0 0) 0x7effd806cad0 con 0x7effd80a30a0 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effd80a6890 con 0x7effd80a30a0 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7effd0003cd0 con 0x7effd80a30a0 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient(hunting): dump: 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.061 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.064 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.064 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 180135097 0 0) 0x7effd8123120 con 0x7effd80a25b0 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 >> v1:172.21.15.94:6791/0 conn(0x7effd8071020 legacy=0x7effd8134750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 >> v1:172.21.15.152:6789/0 conn(0x7effd80a30a0 legacy=0x7effd81cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 monclient: found mon.b 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd80a6320 con 0x7effd80a25b0 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effdcbe9640 5 monclient: authenticate success, global_id 4250 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effdcbe9640 10 monclient: _renew_subs 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effdcbe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:34.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.013+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd8122880 con 0x7effd80a25b0 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effdcbe9640 10 monclient: _renew_subs 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effdcbe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effd8134e60 con 0x7effd80a25b0 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effcc004a90 con 0x7effd80a25b0 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effd4d99640 4 set_mon_vals no callback set 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7effcc004f40 con 0x7effd80a25b0 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effc6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.014+0000 7effc6ffd640 10 monclient: dump: 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.067 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.068 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.015+0000 7effdcbe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.015+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.94:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7effbc000f10 con 0x7effd80a25b0 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.015+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7effcc005120 con 0x7effd80a25b0 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.015+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7effcc045ed0 con 0x7effd80a25b0 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.016+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.152:6801/1475699231 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7effb8044140 con 0x7effb8040490 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.016+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (unknown 3847591702 0 0) 0x7effcc0462c0 con 0x7effd80a25b0 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.016+0000 7effc6ffd640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.016+0000 7effd659c640 10 monclient: get_auth_request con 0x7effb80398a0 auth_method 0 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.016+0000 7effd6d9d640 10 monclient: get_auth_request con 0x7effb8040490 auth_method 0 2024-09-16T10:29:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.018+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== osd.1 v1:172.21.15.94:6800/782646670 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f9030044260 con 0x7f90300405b0 2024-09-16T10:29:34.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.021+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== osd.4 v1:172.21.15.152:6801/1475699231 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7effb8044140 con 0x7effb8040490 2024-09-16T10:29:34.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.027+0000 7f9e87bbc640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.027+0000 7f9e87bbc640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.027+0000 7f9e87bbc640 10 monclient: monmap: 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.028624+0000 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.028624+0000 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.071 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.072 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.072 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.072 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.072 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.072 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.072 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.028+0000 7f9e87bbc640 1 Processor -- start 2024-09-16T10:29:34.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.028+0000 7ff9e2ba3640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.028+0000 7ff9e2ba3640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.028+0000 7f9e87bbc640 1 -- start start 2024-09-16T10:29:34.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.028+0000 7f9e87bbc640 10 monclient: init 2024-09-16T10:29:34.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.028+0000 7ff9e2ba3640 10 monclient: monmap: 2024-09-16T10:29:34.073 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.029551+0000 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.029551+0000 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.074 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.075 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.075 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.029+0000 7f9e87bbc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.029+0000 7ff9e2ba3640 1 Processor -- start 2024-09-16T10:29:34.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.029+0000 7ff9e2ba3640 1 -- start start 2024-09-16T10:29:34.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.029+0000 7f9e87bbc640 10 monclient: _add_conns ranks=[6,4,0,8,7,2,5,3,1] 2024-09-16T10:29:34.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.029+0000 7f9e87bbc640 10 monclient(hunting): picked mon.noname-d con 0x7f9e800a2140 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.029+0000 7ff9e2ba3640 10 monclient: init 2024-09-16T10:29:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.029+0000 7f9e87bbc640 10 monclient(hunting): picked mon.noname-c con 0x7f9e800a15e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7f9e85157640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f9e800a15e0 0x7f9e8013b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33952/0 (socket says 172.21.15.94:33952) 2024-09-16T10:29:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7f9e85157640 1 -- 172.21.15.94:0/2943619300 learned_addr learned my addr 172.21.15.94:0/2943619300 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7f9e87bbc640 10 monclient(hunting): picked mon.noname-a con 0x7f9e800a19f0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7f9e87bbc640 1 -- 172.21.15.94:0/2943619300 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e80061ae0 con 0x7f9e800a19f0 2024-09-16T10:29:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7f9e87bbc640 1 -- 172.21.15.94:0/2943619300 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e800a0920 con 0x7f9e800a15e0 2024-09-16T10:29:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7f9e87bbc640 1 -- 172.21.15.94:0/2943619300 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e800a0360 con 0x7f9e800a2140 2024-09-16T10:29:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7f9e87bbc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7f9e87bbc640 10 monclient(hunting): authenticate will time out at 741.370516s 2024-09-16T10:29:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 10 monclient: _add_conns ranks=[8,6,0,7,4,2,3,5,1] 2024-09-16T10:29:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 10 monclient(hunting): picked mon.noname-e con 0x7ff9dc0a2140 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 10 monclient(hunting): picked mon.noname-d con 0x7ff9dc0a15e0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 10 monclient(hunting): picked mon.noname-a con 0x7ff9dc0a19f0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:34.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc061ae0 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc0a0920 con 0x7ff9dc0a15e0 2024-09-16T10:29:34.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc0a0360 con 0x7ff9dc0a2140 2024-09-16T10:29:34.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e2ba3640 10 monclient(hunting): authenticate will time out at 741.370689s 2024-09-16T10:29:34.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e093f640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7ff9dc0a2140 0x7ff9dc0a2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:58258/0 (socket says 172.21.15.94:58258) 2024-09-16T10:29:34.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.030+0000 7ff9e093f640 1 -- 172.21.15.94:0/531167003 learned_addr learned my addr 172.21.15.94:0/531167003 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e85157640 1 -- 172.21.15.94:0/2943619300 >> v1:172.21.15.94:6791/0 conn(0x7f9e800a15e0 legacy=0x7f9e8013b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e86159640 1 -- 172.21.15.94:0/2943619300 >> v1:172.21.15.94:6789/0 conn(0x7f9e800a19f0 legacy=0x7f9e8013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348759765 0 0) 0x7f9e80061ae0 con 0x7f9e800a19f0 2024-09-16T10:29:34.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 10 monclient(hunting): my global_id is 4275 2024-09-16T10:29:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e6c003520 con 0x7f9e800a19f0 2024-09-16T10:29:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401353959 0 0) 0x7f9e800a0920 con 0x7f9e800a15e0 2024-09-16T10:29:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 10 monclient(hunting): my global_id is 4243 2024-09-16T10:29:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e80061ae0 con 0x7f9e800a15e0 2024-09-16T10:29:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.032+0000 7ff9e093f640 1 -- 172.21.15.94:0/531167003 >> v1:172.21.15.94:6793/0 conn(0x7ff9dc0a2140 legacy=0x7ff9dc0a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7ff9dbfff640 1 -- 172.21.15.94:0/531167003 >> v1:172.21.15.94:6792/0 conn(0x7ff9dc0a15e0 legacy=0x7ff9dc13b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1987522109 0 0) 0x7f9e6c003520 con 0x7f9e800a19f0 2024-09-16T10:29:34.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 >> v1:172.21.15.94:6792/0 conn(0x7f9e800a2140 legacy=0x7f9e800a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:29:34.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 >> v1:172.21.15.94:6791/0 conn(0x7f9e800a15e0 legacy=0x7f9e8013b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7ff9db7fe640 1 -- 172.21.15.94:0/531167003 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143291205 0 0) 0x7ff9dc0a0360 con 0x7ff9dc0a2140 2024-09-16T10:29:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7ff9db7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7ff9db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7ff9db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7ff9db7fe640 10 monclient(hunting): my global_id is 4274 2024-09-16T10:29:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7ff9db7fe640 1 -- 172.21.15.94:0/531167003 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9cc003520 con 0x7ff9dc0a2140 2024-09-16T10:29:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 1 monclient: found mon.noname-a 2024-09-16T10:29:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e800a0640 con 0x7f9e800a19f0 2024-09-16T10:29:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9e78001e30 con 0x7f9e800a19f0 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e87bbc640 5 monclient: authenticate success, global_id 4275 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e87bbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 10 monclient: dump: 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e78003990 con 0x7f9e800a19f0 2024-09-16T10:29:34.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e87bbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e84956640 1 -- 172.21.15.94:0/2943619300 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9e78003e40 con 0x7f9e800a19f0 2024-09-16T10:29:34.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e87bbc640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:34.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.033+0000 7f9e87bbc640 4 set_mon_vals no callback set 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 1 -- 172.21.15.94:0/531167003 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1889729363 0 0) 0x7ff9cc003520 con 0x7ff9dc0a2140 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 1 -- 172.21.15.94:0/531167003 >> v1:172.21.15.94:6792/0 conn(0x7ff9dc0a15e0 legacy=0x7ff9dc13b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 1 -- 172.21.15.94:0/531167003 >> v1:172.21.15.94:6789/0 conn(0x7ff9dc0a19f0 legacy=0x7ff9dc13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 1 monclient: found mon.noname-e 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 1 -- 172.21.15.94:0/531167003 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc0a0640 con 0x7ff9dc0a2140 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 1 -- 172.21.15.94:0/531167003 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff9d0001a60 con 0x7ff9dc0a2140 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: dump: 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.090 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.091 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.091 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.091 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.091 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.091 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.091 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.091 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9e2ba3640 5 monclient: authenticate success, global_id 4274 2024-09-16T10:29:34.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9e2ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7f9e84956640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7f9e84956640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:34.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 1 -- 172.21.15.94:0/531167003 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9d0001d70 con 0x7ff9dc0a2140 2024-09-16T10:29:34.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 1 -- 172.21.15.94:0/531167003 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff9d0004c20 con 0x7ff9dc0a2140 2024-09-16T10:29:34.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:34.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7f9e84956640 10 monclient: dump: 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.093 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.094 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.094 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.094 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.094 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.094 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9db7fe640 10 monclient: dump: 2024-09-16T10:29:34.094 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.094 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.095 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.096 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9e2ba3640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:34.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7ff9e2ba3640 4 set_mon_vals no callback set 2024-09-16T10:29:34.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7f9e87bbc640 10 monclient: shutdown 2024-09-16T10:29:34.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7f9e87bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.034+0000 7f9e87bbc640 1 -- 172.21.15.94:0/2943619300 >> v1:172.21.15.94:6789/0 conn(0x7f9e800a19f0 legacy=0x7f9e8013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7ff9e2ba3640 10 monclient: shutdown 2024-09-16T10:29:34.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7ff9e2ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/531167003 >> v1:172.21.15.94:6793/0 conn(0x7ff9dc0a2140 legacy=0x7ff9dc0a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7f9e87bbc640 1 -- 172.21.15.94:0/2943619300 shutdown_connections 2024-09-16T10:29:34.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/531167003 shutdown_connections 2024-09-16T10:29:34.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/531167003 >> 172.21.15.94:0/531167003 conn(0x7ff9dc0a93c0 msgr2=0x7ff9dc0a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7f9e87bbc640 1 -- 172.21.15.94:0/2943619300 >> 172.21.15.94:0/2943619300 conn(0x7f9e800a93c0 msgr2=0x7f9e800a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7f9e87bbc640 1 -- 172.21.15.94:0/2943619300 shutdown_connections 2024-09-16T10:29:34.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/531167003 shutdown_connections 2024-09-16T10:29:34.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/531167003 wait complete. 2024-09-16T10:29:34.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.035+0000 7f9e87bbc640 1 -- 172.21.15.94:0/2943619300 wait complete. 2024-09-16T10:29:34.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.036+0000 7ff9e2ba3640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.036+0000 7ff9e2ba3640 10 monclient: monmap: 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.100 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.101 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.101 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.101 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.036+0000 7f9e87bbc640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.036+0000 7f9e87bbc640 10 monclient: monmap: 2024-09-16T10:29:34.101 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.101 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.102 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.036+0000 7ff9e2ba3640 1 Processor -- start 2024-09-16T10:29:34.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.036+0000 7ff9e2ba3640 1 -- start start 2024-09-16T10:29:34.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.036+0000 7ff9e2ba3640 10 monclient: init 2024-09-16T10:29:34.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7f9e87bbc640 1 Processor -- start 2024-09-16T10:29:34.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7f9e87bbc640 1 -- start start 2024-09-16T10:29:34.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7f9e87bbc640 10 monclient: init 2024-09-16T10:29:34.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7ff9e2ba3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7ff9e2ba3640 10 monclient: _add_conns ranks=[5,3,6,4,1,8,0,2,7] 2024-09-16T10:29:34.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7ff9e2ba3640 10 monclient(hunting): picked mon.noname-h con 0x7ff9dc0a15e0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:34.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7ff9e2ba3640 10 monclient(hunting): picked mon.noname-g con 0x7ff9dc0a19f0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:34.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7ff9e1140640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7ff9dc13dc00 0x7ff9dc1d7b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:44354/0 (socket says 172.21.15.94:44354) 2024-09-16T10:29:34.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7ff9e1140640 1 -- 172.21.15.94:0/1900806412 learned_addr learned my addr 172.21.15.94:0/1900806412 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7ff9e2ba3640 10 monclient(hunting): picked mon.noname-d con 0x7ff9dc13dc00 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.037+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc0a7440 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 --> v1:172.21.15.94:6800/782646670 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f904c1351d0 con 0x7f90300405b0 2024-09-16T10:29:34.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc138fd0 con 0x7ff9dc0a15e0 2024-09-16T10:29:34.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc06dcb0 con 0x7ff9dc13dc00 2024-09-16T10:29:34.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7ff9e2ba3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7ff9e2ba3640 10 monclient(hunting): authenticate will time out at 741.378070s 2024-09-16T10:29:34.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f90427fc640 1 -- 172.21.15.94:0/2761075692 <== osd.1 v1:172.21.15.94:6800/782646670 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f904c1351d0 con 0x7f90300405b0 2024-09-16T10:29:34.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f9e87bbc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f9e87bbc640 10 monclient: _add_conns ranks=[7,4,8,3,5,2,1,6,0] 2024-09-16T10:29:34.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f9e87bbc640 10 monclient(hunting): picked mon.noname-i con 0x7f9e800a15e0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 >> v1:172.21.15.94:6800/782646670 conn(0x7f90300405b0 legacy=0x7f90300429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f9e85958640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7f9e800a15e0 0x7f9e8013f9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.94:35774/0 (socket says 172.21.15.94:35774) 2024-09-16T10:29:34.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 >> v1:172.21.15.152:6800/2686360781 conn(0x7f90300399e0 legacy=0x7f903003bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f9e85157640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f9e800a19f0 0x7f9e801cfa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33958/0 (socket says 172.21.15.94:33958) 2024-09-16T10:29:34.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f905423a640 10 monclient: shutdown 2024-09-16T10:29:34.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f9e87bbc640 10 monclient(hunting): picked mon.noname-c con 0x7f9e800a19f0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:34.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f905423a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f9e85958640 1 -- 172.21.15.94:0/133973428 learned_addr learned my addr 172.21.15.94:0/133973428 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 >> v1:172.21.15.152:6792/0 conn(0x7f904c071020 legacy=0x7f904c134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.038+0000 7f90527d7640 1 -- 172.21.15.94:0/2761075692 reap_dead start 2024-09-16T10:29:34.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.039+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 shutdown_connections 2024-09-16T10:29:34.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.039+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 >> 172.21.15.94:0/2761075692 conn(0x7f904c0a7290 msgr2=0x7f904c0a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.039+0000 7f9e87bbc640 10 monclient(hunting): picked mon.noname-e con 0x7f9e800a2140 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:34.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.039+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 shutdown_connections 2024-09-16T10:29:34.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.039+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8009efb0 con 0x7f9e800a19f0 2024-09-16T10:29:34.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.039+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e80126a70 con 0x7f9e800a15e0 2024-09-16T10:29:34.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.039+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e800a7620 con 0x7f9e800a2140 2024-09-16T10:29:34.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.039+0000 7f9e87bbc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.039+0000 7f9e87bbc640 10 monclient(hunting): authenticate will time out at 741.379315s 2024-09-16T10:29:34.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7f905423a640 1 -- 172.21.15.94:0/2761075692 wait complete. 2024-09-16T10:29:34.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9e1140640 1 -- 172.21.15.94:0/1900806412 >> v1:172.21.15.94:6792/0 conn(0x7ff9dc13dc00 legacy=0x7ff9dc1d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9dbfff640 1 -- 172.21.15.94:0/1900806412 >> v1:172.21.15.152:6790/0 conn(0x7ff9dc0a19f0 legacy=0x7ff9dc1d43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148728263 0 0) 0x7ff9dc06dcb0 con 0x7ff9dc13dc00 2024-09-16T10:29:34.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 10 monclient(hunting): my global_id is 4209 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c0003520 con 0x7ff9dc13dc00 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9e093f640 1 -- 172.21.15.94:0/1900806412 >> v1:172.21.15.152:6791/0 conn(0x7ff9dc0a15e0 legacy=0x7ff9dc137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151876139 0 0) 0x7ff9dc0a7440 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 10 monclient(hunting): my global_id is 4323 2024-09-16T10:29:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff20fc5a640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff20fc5a640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff20fc5a640 10 monclient: monmap: 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.041442+0000 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.041442+0000 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.116 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9dc06dcb0 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.040+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307157505 0 0) 0x7ff9dc138fd0 con 0x7ff9dc0a15e0 2024-09-16T10:29:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e85958640 1 -- 172.21.15.94:0/133973428 >> v1:172.21.15.152:6792/0 conn(0x7f9e800a15e0 legacy=0x7f9e8013f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): my global_id is 4289 2024-09-16T10:29:34.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9dc0a7440 con 0x7ff9dc0a15e0 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3067607705 0 0) 0x7ff9c0003520 con 0x7ff9dc13dc00 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9dc138fd0 con 0x7ff9dc13dc00 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff9d40046b0 con 0x7ff9dc13dc00 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): dump: 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 537516914 0 0) 0x7ff9dc06dcb0 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e85157640 1 -- 172.21.15.94:0/133973428 >> v1:172.21.15.94:6791/0 conn(0x7f9e800a19f0 legacy=0x7f9e801cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e86159640 1 -- 172.21.15.94:0/133973428 >> v1:172.21.15.94:6793/0 conn(0x7f9e800a2140 legacy=0x7f9e801d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9c0003520 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff9c8001c90 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456225329 0 0) 0x7f9e8009efb0 con 0x7f9e800a19f0 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): my global_id is 4252 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e64003520 con 0x7f9e800a19f0 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84025367 0 0) 0x7f9e80126a70 con 0x7f9e800a15e0 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): my global_id is 4291 2024-09-16T10:29:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e8009efb0 con 0x7f9e800a15e0 2024-09-16T10:29:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031742560 0 0) 0x7f9e800a7620 con 0x7f9e800a2140 2024-09-16T10:29:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 10 monclient(hunting): my global_id is 4283 2024-09-16T10:29:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e80126a70 con 0x7f9e800a2140 2024-09-16T10:29:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient(hunting): dump: 2024-09-16T10:29:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2022169365 0 0) 0x7ff9c0003520 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3312262 0 0) 0x7f9e8009efb0 con 0x7f9e800a15e0 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e800a7620 con 0x7f9e800a15e0 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9e7c0042e0 con 0x7f9e800a15e0 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 10 monclient(hunting): dump: 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.126 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3581458426 0 0) 0x7f9e64003520 con 0x7f9e800a19f0 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e8009efb0 con 0x7f9e800a19f0 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3460714397 0 0) 0x7f9e80126a70 con 0x7f9e800a2140 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e64003520 con 0x7f9e800a2140 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2059041163 0 0) 0x7f9e800a7620 con 0x7f9e800a15e0 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 >> v1:172.21.15.94:6793/0 conn(0x7f9e800a2140 legacy=0x7f9e801d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 >> v1:172.21.15.94:6791/0 conn(0x7f9e800a19f0 legacy=0x7f9e801cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 monclient: found mon.i 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e80127310 con 0x7f9e800a15e0 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7f9e767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 >> v1:172.21.15.94:6792/0 conn(0x7ff9dc13dc00 legacy=0x7ff9dc1d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 >> v1:172.21.15.152:6791/0 conn(0x7ff9dc0a15e0 legacy=0x7ff9dc137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.041+0000 7ff9d97fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 1 monclient: found mon.g 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc06df90 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9e2ba3640 5 monclient: authenticate success, global_id 4323 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9e2ba3640 10 monclient: _renew_subs 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9e2ba3640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9dc06d970 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9e2ba3640 10 monclient: _renew_subs 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9e2ba3640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9dc138640 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9c80039e0 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff9c8003e90 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9d97fa640 10 monclient: dump: 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.131 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.132 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff9daffd640 4 set_mon_vals no callback set 2024-09-16T10:29:34.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff20fc5a640 1 Processor -- start 2024-09-16T10:29:34.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff20fc5a640 1 -- start start 2024-09-16T10:29:34.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.042+0000 7ff20fc5a640 10 monclient: init 2024-09-16T10:29:34.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e7c001180 con 0x7f9e800a15e0 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e77fff640 4 set_mon_vals no callback set 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9e7c001630 con 0x7f9e800a15e0 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e767fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e767fc640 10 monclient: dump: 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.133 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 10 monclient: _add_conns ranks=[7,0,1,5,8,4,3,2,6] 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 10 monclient(hunting): picked mon.noname-i con 0x7ff2080a2140 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 10 monclient(hunting): picked mon.noname-a con 0x7ff2080a15e0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 10 monclient(hunting): picked mon.noname-f con 0x7ff2080a19f0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff208061ae0 con 0x7ff2080a15e0 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080a0920 con 0x7ff2080a19f0 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080a0360 con 0x7ff2080a2140 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20fc5a640 10 monclient(hunting): authenticate will time out at 741.383321s 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e87bbc640 5 monclient: authenticate success, global_id 4291 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e87bbc640 10 monclient: _renew_subs 2024-09-16T10:29:34.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e87bbc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e801400c0 con 0x7f9e800a15e0 2024-09-16T10:29:34.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20d1f5640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff2080a15e0 0x7ff20813b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59388/0 (socket says 172.21.15.94:59388) 2024-09-16T10:29:34.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff20d1f5640 1 -- 172.21.15.94:0/1629805945 learned_addr learned my addr 172.21.15.94:0/1629805945 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e87bbc640 10 monclient: _renew_subs 2024-09-16T10:29:34.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e87bbc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e80125da0 con 0x7f9e800a15e0 2024-09-16T10:29:34.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7ff9c8004030 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff9e2ba3640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:34.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.152:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff9bc000f10 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7ff9c8045e30 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.043+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.94:6805/3102113414 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9c0044150 con 0x7ff9c00404a0 2024-09-16T10:29:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== mon.3 v1:172.21.15.152:6790/0 9 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (unknown 3847591702 0 0) 0x7ff9c8015310 con 0x7ff9dc0a19f0 2024-09-16T10:29:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7ff9d97fa640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T10:29:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7ff9e093f640 10 monclient: get_auth_request con 0x7ff9c00398b0 auth_method 0 2024-09-16T10:29:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7ff9e1140640 10 monclient: get_auth_request con 0x7ff9c00404a0 auth_method 0 2024-09-16T10:29:34.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f9e7c001790 con 0x7f9e800a15e0 2024-09-16T10:29:34.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7f9e87bbc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.152:6792/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e8009f590 con 0x7f9e800a15e0 2024-09-16T10:29:34.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f9e7c045c90 con 0x7f9e800a15e0 2024-09-16T10:29:34.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.94:6804/4202994153 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e64044080 con 0x7f9e640403d0 2024-09-16T10:29:34.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (unknown 3847591702 0 0) 0x7f9e7c014f40 con 0x7f9e800a15e0 2024-09-16T10:29:34.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.044+0000 7f9e767fc640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T10:29:34.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.045+0000 7f9e86159640 10 monclient: get_auth_request con 0x7f9e640403d0 auth_method 0 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.045+0000 7f9f10e19640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.045+0000 7f9f10e19640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.045+0000 7f9f10e19640 10 monclient: monmap: 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.046125+0000 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.046125+0000 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.140 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.141 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.141 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.141 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.141 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.141 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.141 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.141 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.141 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.141 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.045+0000 7ff20d1f5640 1 -- 172.21.15.94:0/1629805945 >> v1:172.21.15.94:6789/0 conn(0x7ff2080a15e0 legacy=0x7ff20813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.045+0000 7ff20d9f6640 1 -- 172.21.15.94:0/1629805945 >> v1:172.21.15.152:6792/0 conn(0x7ff2080a2140 legacy=0x7ff2080a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.045+0000 7ff20e1f7640 1 -- 172.21.15.94:0/1629805945 >> v1:172.21.15.152:6789/0 conn(0x7ff2080a19f0 legacy=0x7ff20813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829025531 0 0) 0x7ff208061ae0 con 0x7ff2080a15e0 2024-09-16T10:29:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): my global_id is 4284 2024-09-16T10:29:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f4003520 con 0x7ff2080a15e0 2024-09-16T10:29:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7f9f10e19640 1 Processor -- start 2024-09-16T10:29:34.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101131734 0 0) 0x7ff2080a0920 con 0x7ff2080a19f0 2024-09-16T10:29:34.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): my global_id is 4222 2024-09-16T10:29:34.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff208061ae0 con 0x7ff2080a19f0 2024-09-16T10:29:34.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397619971 0 0) 0x7ff2080a0360 con 0x7ff2080a2140 2024-09-16T10:29:34.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7f9f10e19640 1 -- start start 2024-09-16T10:29:34.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7f9f10e19640 10 monclient: init 2024-09-16T10:29:34.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient(hunting): my global_id is 4300 2024-09-16T10:29:34.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2080a0920 con 0x7ff2080a2140 2024-09-16T10:29:34.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3174350952 0 0) 0x7ff1f4003520 con 0x7ff2080a15e0 2024-09-16T10:29:34.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7f9e85157640 10 monclient: get_auth_request con 0x7f9e640399f0 auth_method 0 2024-09-16T10:29:34.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 >> v1:172.21.15.152:6792/0 conn(0x7ff2080a2140 legacy=0x7ff2080a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 >> v1:172.21.15.152:6789/0 conn(0x7ff2080a19f0 legacy=0x7ff20813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 monclient: found mon.noname-a 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080a0640 con 0x7ff2080a15e0 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20fc5a640 5 monclient: authenticate success, global_id 4284 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20fc5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff204001a60 con 0x7ff2080a15e0 2024-09-16T10:29:34.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient: dump: 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.151 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.151 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.151 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.151 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.151 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.151 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20fc5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2040048a0 con 0x7ff2080a15e0 2024-09-16T10:29:34.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20c9f4640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 10 monclient: _add_conns ranks=[2,3,0,6,8,7,5,1,4] 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 10 monclient(hunting): picked mon.noname-b con 0x7f9f0c0a2670 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 10 monclient(hunting): picked mon.noname-g con 0x7f9f0c09f220 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 10 monclient(hunting): picked mon.noname-a con 0x7f9f0c137600 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c061ae0 con 0x7f9f0c137600 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c137c20 con 0x7f9f0c0a2670 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c0a1060 con 0x7f9f0c09f220 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f10e19640 10 monclient(hunting): authenticate will time out at 741.387172s 2024-09-16T10:29:34.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 --> v1:172.21.15.152:6801/1475699231 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7effbc002b20 con 0x7effb8040490 2024-09-16T10:29:34.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f03fff640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7f9f0c09f220 0x7f9f0c13f870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.94:46972/0 (socket says 172.21.15.94:46972) 2024-09-16T10:29:34.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7f9f03fff640 1 -- 172.21.15.94:0/392211786 learned_addr learned my addr 172.21.15.94:0/392211786 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20fc5a640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.046+0000 7ff20fc5a640 4 set_mon_vals no callback set 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7ff20c9f4640 1 -- 172.21.15.94:0/1629805945 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff204004d50 con 0x7ff2080a15e0 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7ff20c9f4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7ff20c9f4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7ff20c9f4640 10 monclient: dump: 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.155 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.156 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7effc6ffd640 1 -- 172.21.15.94:0/95169698 <== osd.4 v1:172.21.15.152:6801/1475699231 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7effbc002b20 con 0x7effb8040490 2024-09-16T10:29:34.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7ff20fc5a640 10 monclient: shutdown 2024-09-16T10:29:34.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7ff20fc5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7ff20fc5a640 1 -- 172.21.15.94:0/1629805945 >> v1:172.21.15.94:6789/0 conn(0x7ff2080a15e0 legacy=0x7ff20813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7ff20fc5a640 1 -- 172.21.15.94:0/1629805945 shutdown_connections 2024-09-16T10:29:34.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7ff20fc5a640 1 -- 172.21.15.94:0/1629805945 >> 172.21.15.94:0/1629805945 conn(0x7ff2080a93c0 msgr2=0x7ff2080a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.047+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 >> v1:172.21.15.152:6801/1475699231 conn(0x7effb8040490 legacy=0x7effb80428d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7ff20fc5a640 1 -- 172.21.15.94:0/1629805945 shutdown_connections 2024-09-16T10:29:34.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7ff20fc5a640 1 -- 172.21.15.94:0/1629805945 wait complete. 2024-09-16T10:29:34.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 >> v1:172.21.15.152:6800/2686360781 conn(0x7effb80398a0 legacy=0x7effb803bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7effdcbe9640 10 monclient: shutdown 2024-09-16T10:29:34.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7effdcbe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 >> v1:172.21.15.94:6790/0 conn(0x7effd80a25b0 legacy=0x7effd81cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7effd6d9d640 1 -- 172.21.15.94:0/95169698 reap_dead start 2024-09-16T10:29:34.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 shutdown_connections 2024-09-16T10:29:34.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 >> 172.21.15.94:0/95169698 conn(0x7effd80a7290 msgr2=0x7effd80a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 shutdown_connections 2024-09-16T10:29:34.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7effdcbe9640 1 -- 172.21.15.94:0/95169698 wait complete. 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7ff20fc5a640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.048+0000 7ff20fc5a640 10 monclient: monmap: 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.161 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 1 Processor -- start 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 1 -- start start 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 10 monclient: init 2024-09-16T10:29:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7f9f0ad9d640 1 -- 172.21.15.94:0/392211786 >> v1:172.21.15.94:6789/0 conn(0x7f9f0c137600 legacy=0x7f9f0c142fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7f9f03fff640 1 -- 172.21.15.94:0/392211786 >> v1:172.21.15.152:6790/0 conn(0x7f9f0c09f220 legacy=0x7f9f0c13f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== osd.2 v1:172.21.15.94:6805/3102113414 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7ff9c0044150 con 0x7ff9c00404a0 2024-09-16T10:29:34.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7f9f0a59c640 1 -- 172.21.15.94:0/392211786 >> v1:172.21.15.94:6790/0 conn(0x7f9f0c0a2670 legacy=0x7f9f0c09eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 10 monclient: _add_conns ranks=[1,8,7,5,0,6,4,2,3] 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 10 monclient(hunting): picked mon.noname-f con 0x7ff2080a15e0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 10 monclient(hunting): picked mon.noname-e con 0x7ff2080a19f0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 10 monclient(hunting): picked mon.noname-i con 0x7ff2080a2140 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff208121680 con 0x7ff2080a15e0 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080a72e0 con 0x7ff2080a2140 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20811f580 con 0x7ff2080a19f0 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7ff20fc5a640 10 monclient(hunting): authenticate will time out at 741.389919s 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7ff20e1f7640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7ff2080a2140 0x7ff20813f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.94:35792/0 (socket says 172.21.15.94:35792) 2024-09-16T10:29:34.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7ff20e1f7640 1 -- 172.21.15.94:0/3454508973 learned_addr learned my addr 172.21.15.94:0/3454508973 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== osd.0 v1:172.21.15.94:6804/4202994153 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f9e64044080 con 0x7f9e640403d0 2024-09-16T10:29:34.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7f9f0ad9d640 1 -- 172.21.15.94:0/392211786 >> v1:172.21.15.94:6789/0 conn(0x7f9f0c137600 legacy=0x7f9f0c142fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T10:29:34.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268107041 0 0) 0x7f9f0c0a1060 con 0x7f9f0c09f220 2024-09-16T10:29:34.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient(hunting): my global_id is 4332 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.049+0000 7f9f0ad9d640 1 -- 172.21.15.94:0/392211786 >> v1:172.21.15.94:6789/0 conn(0x7f9f0c137600 legacy=0x7f9f0c142fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eec003520 con 0x7f9f0c09f220 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f0ad9d640 1 --1- 172.21.15.94:0/392211786 >> v1:172.21.15.94:6789/0 conn(0x7f9f0c137600 0x7f9f0c142fa0 :-1 s=OPENED pgs=93 cs=1 l=1).handle_message read tag failed 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f0ad9d640 1 --1- 172.21.15.94:0/392211786 >> v1:172.21.15.94:6789/0 conn(0x7f9f0c137600 0x7f9f0c142fa0 :-1 s=OPENED pgs=93 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.94:6789/0 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814565307 0 0) 0x7f9f0c137c20 con 0x7f9f0c0a2670 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient(hunting): my global_id is 4259 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c0a1060 con 0x7f9f0c0a2670 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1829770490 0 0) 0x7f9eec003520 con 0x7f9f0c09f220 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 >> v1:172.21.15.94:6790/0 conn(0x7f9f0c0a2670 legacy=0x7f9f0c09eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 >> v1:172.21.15.94:6789/0 conn(0x7f9f0c137600 legacy=0x7f9f0c142fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 monclient: found mon.noname-g 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.152:6790/0 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c0a1340 con 0x7f9f0c09f220 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9efc001a60 con 0x7f9f0c09f220 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: dump: 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.169 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f10e19640 5 monclient: authenticate success, global_id 4332 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f10e19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 <== mon.3 v1:172.21.15.152:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9efc001d70 con 0x7f9f0c09f220 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f10e19640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 1 -- 172.21.15.94:0/392211786 <== mon.3 v1:172.21.15.152:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9efc004be0 con 0x7f9f0c09f220 2024-09-16T10:29:34.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f10e19640 4 set_mon_vals no callback set 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.050+0000 7f9f09d9b640 10 monclient: dump: 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.171 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.051+0000 7f9f10e19640 10 monclient: shutdown 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.051+0000 7f9f10e19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.051+0000 7f9f10e19640 1 -- 172.21.15.94:0/392211786 >> v1:172.21.15.152:6790/0 conn(0x7f9f0c09f220 legacy=0x7f9f0c13f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.051+0000 7f9f10e19640 1 -- 172.21.15.94:0/392211786 shutdown_connections 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.051+0000 7f9f10e19640 1 -- 172.21.15.94:0/392211786 >> 172.21.15.94:0/392211786 conn(0x7f9f0c0a1b50 msgr2=0x7f9f0c0a1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.051+0000 7f9f10e19640 1 -- 172.21.15.94:0/392211786 shutdown_connections 2024-09-16T10:29:34.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7f9f10e19640 1 -- 172.21.15.94:0/392211786 wait complete. 2024-09-16T10:29:34.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7ff20e1f7640 1 -- 172.21.15.94:0/3454508973 >> v1:172.21.15.152:6792/0 conn(0x7ff2080a2140 legacy=0x7ff20813f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777524627 0 0) 0x7ff2080a72e0 con 0x7ff2080a2140 2024-09-16T10:29:34.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7ff1fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7ff1fe7fc640 10 monclient(hunting): my global_id is 4309 2024-09-16T10:29:34.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ec003520 con 0x7ff2080a2140 2024-09-16T10:29:34.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7ff20d1f5640 1 -- 172.21.15.94:0/3454508973 >> v1:172.21.15.94:6793/0 conn(0x7ff2080a19f0 legacy=0x7ff20813ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7f9f10e19640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.052+0000 7ff20d9f6640 1 -- 172.21.15.94:0/3454508973 >> v1:172.21.15.152:6789/0 conn(0x7ff2080a15e0 legacy=0x7ff20813e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994458225 0 0) 0x7ff208121680 con 0x7ff2080a15e0 2024-09-16T10:29:34.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): my global_id is 4231 2024-09-16T10:29:34.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2080a72e0 con 0x7ff2080a15e0 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7f9f10e19640 10 monclient: monmap: 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.176 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.177 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.177 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.177 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.177 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.177 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.177 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1300220312 0 0) 0x7ff1ec003520 con 0x7ff2080a2140 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff208121680 con 0x7ff2080a2140 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff1f8003610 con 0x7ff2080a2140 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): dump: 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.178 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:34.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157798794 0 0) 0x7ff20811f580 con 0x7ff2080a19f0 2024-09-16T10:29:34.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): my global_id is 4292 2024-09-16T10:29:34.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ec003520 con 0x7ff2080a19f0 2024-09-16T10:29:34.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7f9f10e19640 1 Processor -- start 2024-09-16T10:29:34.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2411315087 0 0) 0x7ff2080a72e0 con 0x7ff2080a15e0 2024-09-16T10:29:34.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff20811f580 con 0x7ff2080a15e0 2024-09-16T10:29:34.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff200001fb0 con 0x7ff2080a15e0 2024-09-16T10:29:34.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.053+0000 7ff1fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:34.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7f9f10e19640 1 -- start start 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient(hunting): dump: 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.182 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1153289165 0 0) 0x7ff208121680 con 0x7ff2080a2140 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 >> v1:172.21.15.94:6793/0 conn(0x7ff2080a19f0 legacy=0x7ff20813ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 >> v1:172.21.15.152:6789/0 conn(0x7ff2080a15e0 legacy=0x7ff20813e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 1 monclient: found mon.i 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2081213a0 con 0x7ff2080a2140 2024-09-16T10:29:34.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff20fc5a640 5 monclient: authenticate success, global_id 4309 2024-09-16T10:29:34.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff20fc5a640 10 monclient: _renew_subs 2024-09-16T10:29:34.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff20fc5a640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2080a7830 con 0x7ff2080a2140 2024-09-16T10:29:34.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7f9f10e19640 10 monclient: init 2024-09-16T10:29:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff20fc5a640 10 monclient: _renew_subs 2024-09-16T10:29:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff20fc5a640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff208120de0 con 0x7ff2080a2140 2024-09-16T10:29:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f8003bb0 con 0x7ff2080a2140 2024-09-16T10:29:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff1f8004060 con 0x7ff2080a2140 2024-09-16T10:29:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fe7fc640 10 monclient: dump: 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.187 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.187 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7f9f10e19640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7f9f10e19640 10 monclient: _add_conns ranks=[4,6,3,1,8,5,7,0,2] 2024-09-16T10:29:34.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.054+0000 7ff1fffff640 4 set_mon_vals no callback set 2024-09-16T10:29:34.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7ff20fc5a640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6792/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff1f0000f10 con 0x7ff2080a2140 2024-09-16T10:29:34.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7f9f0a59c640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f9f0c0a2670 0x7f9f0c136460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33960/0 (socket says 172.21.15.94:33960) 2024-09-16T10:29:34.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7ff1f80041c0 con 0x7ff2080a2140 2024-09-16T10:29:34.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7f9f0a59c640 1 -- 172.21.15.94:0/1374351252 learned_addr learned my addr 172.21.15.94:0/1374351252 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7f9f10e19640 10 monclient(hunting): picked mon.noname-c con 0x7f9f0c0a2670 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:34.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7ff1f8046050 con 0x7ff2080a2140 2024-09-16T10:29:34.190 INFO:teuthology.orchestra.run.smithi094.stdout:47244640259 2024-09-16T10:29:34.190 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:29:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7f9f10e19640 10 monclient(hunting): picked mon.noname-d con 0x7f9f0c137600 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7f9f10e19640 10 monclient(hunting): picked mon.noname-g con 0x7f9f0c136b70 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6805/823649059 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1ec0441a0 con 0x7ff1ec0404f0 2024-09-16T10:29:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (unknown 3847591702 0 0) 0x7ff1f8046440 con 0x7ff2080a2140 2024-09-16T10:29:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c070b30 con 0x7f9f0c136b70 2024-09-16T10:29:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7ff1fe7fc640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T10:29:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.055+0000 7ff20d9f6640 10 monclient: get_auth_request con 0x7ff1ec039900 auth_method 0 2024-09-16T10:29:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.056+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c0a4980 con 0x7f9f0c0a2670 2024-09-16T10:29:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.056+0000 7ff20d1f5640 10 monclient: get_auth_request con 0x7ff1ec0404f0 auth_method 0 2024-09-16T10:29:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.056+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c071f70 con 0x7f9f0c137600 2024-09-16T10:29:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.056+0000 7f9f10e19640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.056+0000 7f9f10e19640 10 monclient(hunting): authenticate will time out at 741.396116s 2024-09-16T10:29:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.057+0000 7f9f0a59c640 1 -- 172.21.15.94:0/1374351252 >> v1:172.21.15.94:6791/0 conn(0x7f9f0c0a2670 legacy=0x7f9f0c136460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.057+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487287708 0 0) 0x7f9f0c0a4980 con 0x7f9f0c0a2670 2024-09-16T10:29:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.057+0000 7f9f02ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.057+0000 7f9f02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.057+0000 7f9f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.057+0000 7f9f02ffd640 10 monclient(hunting): my global_id is 4261 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.057+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef0003520 con 0x7f9f0c0a2670 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.058+0000 7f9f03fff640 1 -- 172.21.15.94:0/1374351252 >> v1:172.21.15.94:6792/0 conn(0x7f9f0c137600 legacy=0x7f9f0c1d41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.058+0000 7f9f0ad9d640 1 -- 172.21.15.94:0/1374351252 >> v1:172.21.15.152:6790/0 conn(0x7f9f0c136b70 legacy=0x7f9f0c1d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.058+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 DispatchQueue::entry inject delay of 0.002000 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025045493 0 0) 0x7f9f0c070b30 con 0x7f9f0c136b70 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): my global_id is 4341 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c0a4980 con 0x7f9f0c136b70 2024-09-16T10:29:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3890072571 0 0) 0x7f9ef0003520 con 0x7f9f0c0a2670 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f0c070b30 con 0x7f9f0c0a2670 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782051609 0 0) 0x7f9f0c071f70 con 0x7f9f0c137600 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): my global_id is 4218 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef0003520 con 0x7f9f0c137600 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9ef4003610 con 0x7f9f0c0a2670 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): dump: 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1226744401 0 0) 0x7f9f0c0a4980 con 0x7f9f0c136b70 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f0c071f70 con 0x7f9f0c136b70 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3449252002 0 0) 0x7f9ef0003520 con 0x7f9f0c137600 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f0c0a4980 con 0x7f9f0c137600 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9f04002100 con 0x7f9f0c136b70 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient(hunting): dump: 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2095139809 0 0) 0x7f9f0c070b30 con 0x7f9f0c0a2670 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 >> v1:172.21.15.94:6792/0 conn(0x7f9f0c137600 legacy=0x7f9f0c1d41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 >> v1:172.21.15.152:6790/0 conn(0x7f9f0c136b70 legacy=0x7f9f0c1d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 monclient: found mon.c 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c06fef0 con 0x7f9f0c0a2670 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ef4003b50 con 0x7f9f0c0a2670 2024-09-16T10:29:34.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f02ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.060+0000 7f9f0959a640 4 set_mon_vals no callback set 2024-09-16T10:29:34.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== osd.3 v1:172.21.15.152:6805/823649059 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7ff1ec0441a0 con 0x7ff1ec0404f0 2024-09-16T10:29:34.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f10e19640 5 monclient: authenticate success, global_id 4261 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9ef4004000 con 0x7f9f0c0a2670 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f10e19640 10 monclient: _renew_subs 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f10e19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0c09db80 con 0x7f9f0c0a2670 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f02ffd640 10 monclient: dump: 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f10e19640 10 monclient: _renew_subs 2024-09-16T10:29:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f10e19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.061+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0c070840 con 0x7f9f0c0a2670 2024-09-16T10:29:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.062+0000 7f9f10e19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.062+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.94:6791/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f0c04e730 con 0x7f9f0c0a2670 2024-09-16T10:29:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.063+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f9ef40041e0 con 0x7f9f0c0a2670 2024-09-16T10:29:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.063+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f9ef4045e80 con 0x7f9f0c0a2670 2024-09-16T10:29:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.063+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.152:6806/3657457384 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9efc001320 con 0x7f9ef0040840 2024-09-16T10:29:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.063+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (unknown 3847591702 0 0) 0x7f9ef40461b0 con 0x7f9f0c0a2670 2024-09-16T10:29:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.063+0000 7f9f02ffd640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T10:29:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.063+0000 7f9f0ad9d640 10 monclient: get_auth_request con 0x7f9ef0040840 auth_method 0 2024-09-16T10:29:34.207 INFO:teuthology.orchestra.run.smithi094.stdout:47244640259 2024-09-16T10:29:34.207 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T10:29:34.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.064+0000 7f9f03fff640 10 monclient: get_auth_request con 0x7f9ef0039d40 auth_method 0 2024-09-16T10:29:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.068+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== osd.5 v1:172.21.15.152:6806/3657457384 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f9efc001320 con 0x7f9ef0040840 2024-09-16T10:29:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.068+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 --> v1:172.21.15.94:6804/4202994153 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9e8004e730 con 0x7f9e640403d0 2024-09-16T10:29:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 --> v1:172.21.15.94:6805/3102113414 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff9bc002b20 con 0x7ff9c00404a0 2024-09-16T10:29:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e767fc640 1 -- 172.21.15.94:0/133973428 <== osd.0 v1:172.21.15.94:6804/4202994153 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f9e78002040 con 0x7f9e640403d0 2024-09-16T10:29:34.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9d97fa640 1 -- 172.21.15.94:0/1900806412 <== osd.2 v1:172.21.15.94:6805/3102113414 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 839772338) 0x7ff9bc002b20 con 0x7ff9c00404a0 2024-09-16T10:29:34.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 >> v1:172.21.15.94:6805/3102113414 conn(0x7ff9c00404a0 legacy=0x7ff9c00428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 >> v1:172.21.15.94:6804/4202994153 conn(0x7f9e640403d0 legacy=0x7f9e64042810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 >> v1:172.21.15.152:6800/2686360781 conn(0x7f9e640399f0 legacy=0x7f9e6403beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 >> v1:172.21.15.152:6800/2686360781 conn(0x7ff9c00398b0 legacy=0x7ff9c003bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e87bbc640 10 monclient: shutdown 2024-09-16T10:29:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e87bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 >> v1:172.21.15.152:6792/0 conn(0x7f9e800a15e0 legacy=0x7f9e8013f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9e2ba3640 10 monclient: shutdown 2024-09-16T10:29:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9e2ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 >> v1:172.21.15.152:6790/0 conn(0x7ff9dc0a19f0 legacy=0x7ff9dc1d43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9e1140640 1 -- 172.21.15.94:0/1900806412 reap_dead start 2024-09-16T10:29:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e86159640 1 -- 172.21.15.94:0/133973428 reap_dead start 2024-09-16T10:29:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 shutdown_connections 2024-09-16T10:29:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 >> 172.21.15.94:0/133973428 conn(0x7f9e800a93c0 msgr2=0x7f9e800a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 shutdown_connections 2024-09-16T10:29:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 >> 172.21.15.94:0/1900806412 conn(0x7ff9dc0a93c0 msgr2=0x7ff9dc0a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.069+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 shutdown_connections 2024-09-16T10:29:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.070+0000 7f9e87bbc640 1 -- 172.21.15.94:0/133973428 wait complete. 2024-09-16T10:29:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.070+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 shutdown_connections 2024-09-16T10:29:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.070+0000 7ff9e2ba3640 1 -- 172.21.15.94:0/1900806412 wait complete. 2024-09-16T10:29:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.076+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 --> v1:172.21.15.152:6805/823649059 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff1f0002b20 con 0x7ff1ec0404f0 2024-09-16T10:29:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.077+0000 7ff1fe7fc640 1 -- 172.21.15.94:0/3454508973 <== osd.3 v1:172.21.15.152:6805/823649059 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 654699149) 0x7ff1f0002b20 con 0x7ff1ec0404f0 2024-09-16T10:29:34.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 >> v1:172.21.15.152:6805/823649059 conn(0x7ff1ec0404f0 legacy=0x7ff1ec042930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 >> v1:172.21.15.152:6800/2686360781 conn(0x7ff1ec039900 legacy=0x7ff1ec03bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20fc5a640 10 monclient: shutdown 2024-09-16T10:29:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20fc5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 >> v1:172.21.15.152:6792/0 conn(0x7ff2080a2140 legacy=0x7ff20813f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20e1f7640 1 -- 172.21.15.94:0/3454508973 reap_dead start 2024-09-16T10:29:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 shutdown_connections 2024-09-16T10:29:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 >> 172.21.15.94:0/3454508973 conn(0x7ff2080a93c0 msgr2=0x7ff2080a7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 shutdown_connections 2024-09-16T10:29:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.078+0000 7ff20fc5a640 1 -- 172.21.15.94:0/3454508973 wait complete. 2024-09-16T10:29:34.216 INFO:teuthology.orchestra.run.smithi094.stdout:47244640259 2024-09-16T10:29:34.216 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:29:34.216 INFO:teuthology.orchestra.run.smithi094.stdout:51539607555 2024-09-16T10:29:34.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:29:34.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.083+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 --> v1:172.21.15.152:6806/3657457384 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9f0c071a90 con 0x7f9ef0040840 2024-09-16T10:29:34.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f02ffd640 1 -- 172.21.15.94:0/1374351252 <== osd.5 v1:172.21.15.152:6806/3657457384 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 654699149) 0x7f9f0c071a90 con 0x7f9ef0040840 2024-09-16T10:29:34.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 >> v1:172.21.15.152:6806/3657457384 conn(0x7f9ef0040840 legacy=0x7f9ef0042c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 >> v1:172.21.15.152:6800/2686360781 conn(0x7f9ef0039d40 legacy=0x7f9ef003c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f10e19640 10 monclient: shutdown 2024-09-16T10:29:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f10e19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 >> v1:172.21.15.94:6791/0 conn(0x7f9f0c0a2670 legacy=0x7f9f0c136460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f0ad9d640 1 -- 172.21.15.94:0/1374351252 reap_dead start 2024-09-16T10:29:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 shutdown_connections 2024-09-16T10:29:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 >> 172.21.15.94:0/1374351252 conn(0x7f9f0c0a1b50 msgr2=0x7f9f0c09e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 shutdown_connections 2024-09-16T10:29:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.084+0000 7f9f10e19640 1 -- 172.21.15.94:0/1374351252 wait complete. 2024-09-16T10:29:34.219 INFO:teuthology.orchestra.run.smithi094.stdout:47244640260 2024-09-16T10:29:34.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T10:29:34.220 INFO:teuthology.orchestra.run.smithi094.stdout:47244640260 2024-09-16T10:29:34.221 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.354+0000 7f074db18640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.354+0000 7f074db18640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.354+0000 7f074db18640 10 monclient: monmap: 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.355171+0000 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.355171+0000 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.356 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.357 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.357 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.357 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.357 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.357 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.357 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.357 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.357 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.355+0000 7f074db18640 1 Processor -- start 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.355+0000 7f074db18640 1 -- start start 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.355+0000 7f074db18640 10 monclient: init 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.355+0000 7f074db18640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.355+0000 7f074db18640 10 monclient: _add_conns ranks=[3,4,6,7,2,5,8,0,1] 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.355+0000 7f074db18640 10 monclient(hunting): picked mon.noname-g con 0x7f07480a38d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.355+0000 7f074db18640 10 monclient(hunting): picked mon.noname-c con 0x7f074806c670 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.356+0000 7f074db18640 10 monclient(hunting): picked mon.noname-d con 0x7f07480a43c0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.356+0000 7f074db18640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0748061ae0 con 0x7f07480a38d0 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.356+0000 7f074db18640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07480a04d0 con 0x7f074806c670 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.356+0000 7f074db18640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07480a3250 con 0x7f07480a43c0 2024-09-16T10:29:34.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.356+0000 7f074db18640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.356+0000 7f074db18640 10 monclient(hunting): authenticate will time out at 741.696106s 2024-09-16T10:29:34.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.356+0000 7f0746ffd640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f074806c670 0x7f074813bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33966/0 (socket says 172.21.15.94:33966) 2024-09-16T10:29:34.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.356+0000 7f0746ffd640 1 -- 172.21.15.94:0/3767715893 learned_addr learned my addr 172.21.15.94:0/3767715893 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.358+0000 7fe5892d7640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.358+0000 7fe5892d7640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.358+0000 7fe5892d7640 10 monclient: monmap: 2024-09-16T10:29:34.360 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.359206+0000 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.359206+0000 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.362 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.362 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.362 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.362 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.358+0000 7f0746ffd640 1 -- 172.21.15.94:0/3767715893 >> v1:172.21.15.94:6791/0 conn(0x7f074806c670 legacy=0x7f074813bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.358+0000 7f07477fe640 1 -- 172.21.15.94:0/3767715893 >> v1:172.21.15.152:6790/0 conn(0x7f07480a38d0 legacy=0x7f07480a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.358+0000 7f0747fff640 1 -- 172.21.15.94:0/3767715893 >> v1:172.21.15.94:6792/0 conn(0x7f07480a43c0 legacy=0x7f074813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.359+0000 7fe5892d7640 1 Processor -- start 2024-09-16T10:29:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.359+0000 7fe5892d7640 1 -- start start 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.359+0000 7fe5892d7640 10 monclient: init 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.359+0000 7fe5892d7640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.359+0000 7fe5892d7640 10 monclient: _add_conns ranks=[1,2,0,7,3,6,5,4,8] 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.359+0000 7fe5892d7640 10 monclient(hunting): picked mon.noname-f con 0x7fe5840a38d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.359+0000 7fe5892d7640 10 monclient(hunting): picked mon.noname-b con 0x7fe58406c670 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.359+0000 7fe5892d7640 10 monclient(hunting): picked mon.noname-a con 0x7fe5840a43c0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.359+0000 7fe5892d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe584061ae0 con 0x7fe5840a43c0 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.360+0000 7fe5827fc640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fe58406c670 0x7fe58413bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53418/0 (socket says 172.21.15.94:53418) 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.360+0000 7fe5827fc640 1 -- 172.21.15.94:0/3393254319 learned_addr learned my addr 172.21.15.94:0/3393254319 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.360+0000 7fe5892d7640 1 -- 172.21.15.94:0/3393254319 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5840a04d0 con 0x7fe5840a38d0 2024-09-16T10:29:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.360+0000 7fe5892d7640 1 -- 172.21.15.94:0/3393254319 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5840a3250 con 0x7fe58406c670 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.360+0000 7fe5892d7640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.360+0000 7fe5892d7640 10 monclient(hunting): authenticate will time out at 741.700211s 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe5827fc640 1 -- 172.21.15.94:0/3393254319 >> v1:172.21.15.94:6790/0 conn(0x7fe58406c670 legacy=0x7fe58413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe5837fe640 1 -- 172.21.15.94:0/3393254319 >> v1:172.21.15.94:6789/0 conn(0x7fe5840a43c0 legacy=0x7fe58413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe582ffd640 1 -- 172.21.15.94:0/3393254319 >> v1:172.21.15.152:6789/0 conn(0x7fe5840a38d0 legacy=0x7fe5840a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 1 -- 172.21.15.94:0/3393254319 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392995698 0 0) 0x7fe584061ae0 con 0x7fe5840a43c0 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient(hunting): my global_id is 4302 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 1 -- 172.21.15.94:0/3393254319 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe56c003520 con 0x7fe5840a43c0 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 1 -- 172.21.15.94:0/3393254319 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1231394982 0 0) 0x7fe56c003520 con 0x7fe5840a43c0 2024-09-16T10:29:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 1 -- 172.21.15.94:0/3393254319 >> v1:172.21.15.94:6790/0 conn(0x7fe58406c670 legacy=0x7fe58413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 1 -- 172.21.15.94:0/3393254319 >> v1:172.21.15.152:6789/0 conn(0x7fe5840a38d0 legacy=0x7fe5840a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 1 monclient: found mon.noname-a 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 1 -- 172.21.15.94:0/3393254319 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5840a0a00 con 0x7fe5840a43c0 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 1 -- 172.21.15.94:0/3393254319 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe57c001ef0 con 0x7fe5840a43c0 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe581ffb640 10 monclient: dump: 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe5892d7640 5 monclient: authenticate success, global_id 4302 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.362+0000 7fe5892d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe581ffb640 1 -- 172.21.15.94:0/3393254319 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe57c003990 con 0x7fe5840a43c0 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe581ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe5892d7640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe5892d7640 4 set_mon_vals no callback set 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe581ffb640 1 -- 172.21.15.94:0/3393254319 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe57c003f00 con 0x7fe5840a43c0 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe581ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe581ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe581ffb640 10 monclient: dump: 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe5892d7640 10 monclient: shutdown 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe5892d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe5892d7640 1 -- 172.21.15.94:0/3393254319 >> v1:172.21.15.94:6789/0 conn(0x7fe5840a43c0 legacy=0x7fe58413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe5892d7640 1 -- 172.21.15.94:0/3393254319 shutdown_connections 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe5892d7640 1 -- 172.21.15.94:0/3393254319 >> 172.21.15.94:0/3393254319 conn(0x7fe5840a59d0 msgr2=0x7fe5840a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe5892d7640 1 -- 172.21.15.94:0/3393254319 shutdown_connections 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.363+0000 7fe5892d7640 1 -- 172.21.15.94:0/3393254319 wait complete. 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.364+0000 7fe5892d7640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.364+0000 7fe5892d7640 10 monclient: monmap: 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.374 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.364+0000 7fe5892d7640 1 Processor -- start 2024-09-16T10:29:34.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.364+0000 7fe5892d7640 1 -- start start 2024-09-16T10:29:34.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.364+0000 7fe5892d7640 10 monclient: init 2024-09-16T10:29:34.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 10 monclient: _add_conns ranks=[0,4,3,7,5,2,6,8,1] 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 10 monclient(hunting): picked mon.noname-a con 0x7fe58406c670 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe582ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe58406c670 0x7fe5841369c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59414/0 (socket says 172.21.15.94:59414) 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5827fc640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fe5840a38d0 0x7fe5841cfda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33974/0 (socket says 172.21.15.94:33974) 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 10 monclient(hunting): picked mon.noname-c con 0x7fe5840a38d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5827fc640 1 -- 172.21.15.94:0/4097394300 learned_addr learned my addr 172.21.15.94:0/4097394300 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 10 monclient(hunting): picked mon.noname-g con 0x7fe5840a43c0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58406b950 con 0x7fe58406c670 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5841cb9d0 con 0x7fe5840a43c0 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58409f220 con 0x7fe5840a38d0 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.365+0000 7fe5892d7640 10 monclient(hunting): authenticate will time out at 741.705785s 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.367+0000 7fe5827fc640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6791/0 conn(0x7fe5840a38d0 legacy=0x7fe5841cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.367+0000 7fe582ffd640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6789/0 conn(0x7fe58406c670 legacy=0x7fe5841369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560945177 0 0) 0x7fe58406b950 con 0x7fe58406c670 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient(hunting): my global_id is 4311 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe570003520 con 0x7fe58406c670 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe5837fe640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.152:6790/0 conn(0x7fe5840a43c0 legacy=0x7fe5841d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334426177 0 0) 0x7fe570003520 con 0x7fe58406c670 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe58406b950 con 0x7fe58406c670 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe574001c90 con 0x7fe58406c670 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:34.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient(hunting): dump: 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.382 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3163469880 0 0) 0x7fe58406b950 con 0x7fe58406c670 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6791/0 conn(0x7fe5840a38d0 legacy=0x7fe5841cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.152:6790/0 conn(0x7fe5840a43c0 legacy=0x7fe5841d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 monclient: found mon.a 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58409eef0 con 0x7fe58406c670 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.368+0000 7fe56b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe574003aa0 con 0x7fe58406c670 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe56b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe574003f50 con 0x7fe58406c670 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe5892d7640 5 monclient: authenticate success, global_id 4311 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe5892d7640 10 monclient: _renew_subs 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe5892d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5841270d0 con 0x7fe58406c670 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe5817fa640 4 set_mon_vals no callback set 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe5892d7640 10 monclient: _renew_subs 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe5892d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe584125730 con 0x7fe58406c670 2024-09-16T10:29:34.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe56b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe56b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.369+0000 7fe56b7fe640 10 monclient: dump: 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.370+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fe5740041f0 con 0x7fe58406c670 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.370+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7fe574045f70 con 0x7fe58406c670 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.370+0000 7fe5892d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.370+0000 7fe5892d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.370+0000 7fe5892d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.370+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe584069e70 con 0x7fe58406c670 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.371+0000 7fe5827fc640 10 monclient: get_auth_request con 0x7fe5700390d0 auth_method 0 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.375+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fe5740151e0 con 0x7fe58406c670 2024-09-16T10:29:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.375+0000 7fe56b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:34.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.375+0000 7fe56b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:34.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7f73f422b640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7f73f422b640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7f73f422b640 10 monclient: monmap: 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.396852+0000 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.396852+0000 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.398 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.399 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.399 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.399 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.399 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.399 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7fe29d6a3640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7fe29d6a3640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7fe29d6a3640 10 monclient: monmap: 2024-09-16T10:29:34.399 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.397071+0000 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.397071+0000 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.400 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.401 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7f9d4ee09640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7f9d4ee09640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7f9d4ee09640 10 monclient: monmap: 2024-09-16T10:29:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.397043+0000 2024-09-16T10:29:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.397043+0000 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.402 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7f73f422b640 1 Processor -- start 2024-09-16T10:29:34.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.396+0000 7f9d4ee09640 1 Processor -- start 2024-09-16T10:29:34.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 1 -- start start 2024-09-16T10:29:34.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 1 -- start start 2024-09-16T10:29:34.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 10 monclient: init 2024-09-16T10:29:34.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7fe29d6a3640 1 Processor -- start 2024-09-16T10:29:34.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 10 monclient: init 2024-09-16T10:29:34.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7fe29d6a3640 1 -- start start 2024-09-16T10:29:34.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7fe29d6a3640 10 monclient: init 2024-09-16T10:29:34.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 10 monclient: _add_conns ranks=[2,5,6,8,0,7,1,4,3] 2024-09-16T10:29:34.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 10 monclient: _add_conns ranks=[1,2,5,7,6,0,3,4,8] 2024-09-16T10:29:34.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 10 monclient(hunting): picked mon.noname-b con 0x7f9d480a0590 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 10 monclient(hunting): picked mon.noname-h con 0x7f9d48071860 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:34.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 10 monclient(hunting): picked mon.noname-f con 0x7f73ec0a38d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:34.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 10 monclient(hunting): picked mon.noname-b con 0x7f73ec06c670 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 10 monclient(hunting): picked mon.noname-h con 0x7f73ec0a43c0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:34.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec061ae0 con 0x7f73ec0a38d0 2024-09-16T10:29:34.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec0a04d0 con 0x7f73ec06c670 2024-09-16T10:29:34.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec0a3250 con 0x7f73ec0a43c0 2024-09-16T10:29:34.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 10 monclient(hunting): picked mon.noname-d con 0x7f9d480a1080 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f422b640 10 monclient(hunting): authenticate will time out at 741.737686s 2024-09-16T10:29:34.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d48061ae0 con 0x7f9d480a0590 2024-09-16T10:29:34.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d48070910 con 0x7f9d48071860 2024-09-16T10:29:34.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d480715e0 con 0x7f9d480a1080 2024-09-16T10:29:34.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4ee09640 10 monclient(hunting): authenticate will time out at 741.737797s 2024-09-16T10:29:34.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4d3a6640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f9d480a1080 0x7f9d4813efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:44388/0 (socket says 172.21.15.94:44388) 2024-09-16T10:29:34.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f9d4d3a6640 1 -- 172.21.15.94:0/3206866105 learned_addr learned my addr 172.21.15.94:0/3206866105 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f17c6640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f73ec06c670 0x7f73ec13bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53426/0 (socket says 172.21.15.94:53426) 2024-09-16T10:29:34.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.397+0000 7f73f17c6640 1 -- 172.21.15.94:0/2796412556 learned_addr learned my addr 172.21.15.94:0/2796412556 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 10 monclient: _add_conns ranks=[6,3,7,8,0,2,1,4,5] 2024-09-16T10:29:34.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 10 monclient(hunting): picked mon.noname-d con 0x7fe2980a3860 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 10 monclient(hunting): picked mon.noname-g con 0x7fe29806c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:34.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 10 monclient(hunting): picked mon.noname-i con 0x7fe2980a4350 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe298061ae0 con 0x7fe29806c1d0 2024-09-16T10:29:34.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2980a0600 con 0x7fe2980a3860 2024-09-16T10:29:34.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2980a0b30 con 0x7fe2980a4350 2024-09-16T10:29:34.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe2967fc640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7fe29806c1d0 0x7fe29813b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.94:47010/0 (socket says 172.21.15.94:47010) 2024-09-16T10:29:34.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe2967fc640 1 -- 172.21.15.94:0/3705355088 learned_addr learned my addr 172.21.15.94:0/3705355088 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.398+0000 7fe29d6a3640 10 monclient(hunting): authenticate will time out at 741.738476s 2024-09-16T10:29:34.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.399+0000 7f9d4d3a6640 1 -- 172.21.15.94:0/3206866105 >> v1:172.21.15.94:6792/0 conn(0x7f9d480a1080 legacy=0x7f9d4813efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.400+0000 7f9d4cba5640 1 -- 172.21.15.94:0/3206866105 >> v1:172.21.15.94:6790/0 conn(0x7f9d480a0590 legacy=0x7f9d480a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.400+0000 7f73f27c8640 1 -- 172.21.15.94:0/2796412556 >> v1:172.21.15.152:6791/0 conn(0x7f73ec0a43c0 legacy=0x7f73ec13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.400+0000 7f73f1fc7640 1 -- 172.21.15.94:0/2796412556 >> v1:172.21.15.152:6789/0 conn(0x7f73ec0a38d0 legacy=0x7f73ec0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.400+0000 7f9d3ffff640 1 -- 172.21.15.94:0/3206866105 >> v1:172.21.15.152:6791/0 conn(0x7f9d48071860 legacy=0x7f9d4813b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.400+0000 7fe2967fc640 1 -- 172.21.15.94:0/3705355088 >> v1:172.21.15.152:6790/0 conn(0x7fe29806c1d0 legacy=0x7fe29813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.400+0000 7f71c2fed640 10 monclient: get_monmap_and_config 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.400+0000 7f71c2fed640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.400+0000 7f71c2fed640 10 monclient: monmap: 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:34.401381+0000 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:34.401381+0000 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.400+0000 7fe2977fe640 1 -- 172.21.15.94:0/3705355088 >> v1:172.21.15.152:6792/0 conn(0x7fe2980a4350 legacy=0x7fe29813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 1 Processor -- start 2024-09-16T10:29:34.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 1 -- start start 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 10 monclient: init 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 10 monclient: _add_conns ranks=[7,6,2,0,3,8,4,1,5] 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 10 monclient(hunting): picked mon.noname-i con 0x7f71bc0a15a0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 10 monclient(hunting): picked mon.noname-d con 0x7f71bc09fe50 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 10 monclient(hunting): picked mon.noname-b con 0x7f71bc0a2090 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc061ae0 con 0x7f71bc0a2090 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71bbfff640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f71bc09fe50 0x7f71bc139610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:44414/0 (socket says 172.21.15.94:44414) 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71bbfff640 1 -- 172.21.15.94:0/4126769015 learned_addr learned my addr 172.21.15.94:0/4126769015 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.401+0000 7f71c2fed640 1 -- 172.21.15.94:0/4126769015 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc070120 con 0x7f71bc09fe50 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.402+0000 7f71c2fed640 1 -- 172.21.15.94:0/4126769015 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc06abb0 con 0x7f71bc0a15a0 2024-09-16T10:29:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.402+0000 7f71c2fed640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.402+0000 7f71c2fed640 10 monclient(hunting): authenticate will time out at 741.742034s 2024-09-16T10:29:34.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.402+0000 7f73f17c6640 1 -- 172.21.15.94:0/2796412556 >> v1:172.21.15.94:6790/0 conn(0x7f73ec06c670 legacy=0x7f73ec13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.402+0000 7fe296ffd640 1 -- 172.21.15.94:0/3705355088 >> v1:172.21.15.94:6792/0 conn(0x7fe2980a3860 legacy=0x7fe2980a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.404+0000 7f71bbfff640 1 -- 172.21.15.94:0/4126769015 >> v1:172.21.15.94:6792/0 conn(0x7f71bc09fe50 legacy=0x7f71bc139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.404+0000 7f71c158a640 1 -- 172.21.15.94:0/4126769015 >> v1:172.21.15.94:6790/0 conn(0x7f71bc0a2090 legacy=0x7f71bc13cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.404+0000 7f71c0d89640 1 -- 172.21.15.94:0/4126769015 >> v1:172.21.15.152:6792/0 conn(0x7f71bc0a15a0 legacy=0x7f71bc0a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe5892d7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe5892d7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe5892d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe5840008d0 con 0x7fe58406c670 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe582ffd640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6789/0 conn(0x7fe58406c670 legacy=0x7fe5841369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe582ffd640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6789/0 conn(0x7fe58406c670 legacy=0x7fe5841369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe582ffd640 1 --1- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6789/0 conn(0x7fe58406c670 0x7fe5841369c0 :-1 s=OPENED pgs=95 cs=1 l=1).handle_message read tag failed 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe582ffd640 1 --1- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6789/0 conn(0x7fe58406c670 0x7fe5841369c0 :-1 s=OPENED pgs=95 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe56b7fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.94:6789/0 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe56b7fe640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6789/0 conn(0x7fe58406c670 legacy=0x7fe5841369c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe56b7fe640 10 monclient: _add_conns ranks=[7,0,2,4,6,3,1,5,8] 2024-09-16T10:29:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.543+0000 7fe56b7fe640 10 monclient(hunting): picked mon.i con 0x7fe57003fa80 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.544+0000 7fe56b7fe640 10 monclient(hunting): picked mon.a con 0x7fe5700438b0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.544+0000 7fe56b7fe640 10 monclient(hunting): picked mon.b con 0x7fe570047a00 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.544+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe58406b950 con 0x7fe5700438b0 2024-09-16T10:29:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.544+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe570003520 con 0x7fe570047a00 2024-09-16T10:29:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.544+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe57004dbd0 con 0x7fe57003fa80 2024-09-16T10:29:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.544+0000 7fe56b7fe640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe5837fe640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6789/0 conn(0x7fe5700438b0 legacy=0x7fe570045d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe582ffd640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6790/0 conn(0x7fe570047a00 legacy=0x7fe570049ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852240904 0 0) 0x7fe58406b950 con 0x7fe5700438b0 2024-09-16T10:29:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:29:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fe57004cba0 con 0x7fe5700438b0 2024-09-16T10:29:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe582ffd640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.152:6792/0 conn(0x7fe57003fa80 legacy=0x7fe570041e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2466060408 0 0) 0x7fe57004cba0 con 0x7fe5700438b0 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.152:6792/0 conn(0x7fe57003fa80 legacy=0x7fe570041e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6790/0 conn(0x7fe570047a00 legacy=0x7fe570049ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 monclient: found mon.a 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fe57004ee50 con 0x7fe5700438b0 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe58406b950 con 0x7fe5700438b0 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe57c003f90 con 0x7fe5700438b0 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: dump: 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe57c001be0 con 0x7fe5700438b0 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe56b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.546+0000 7fe5817fa640 4 set_mon_vals no callback set 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.547+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fe57c001d50 con 0x7fe5700438b0 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.547+0000 7fe56b7fe640 1 -- 172.21.15.94:0/4097394300 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 555833778) 0x7fe57c044b80 con 0x7fe5700438b0 2024-09-16T10:29:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.547+0000 7fe56b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:34.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.547+0000 7fe56b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:34.553 INFO:teuthology.orchestra.run.smithi094.stdout:47244640258 2024-09-16T10:29:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.152:6800/2686360781 conn(0x7fe5700390d0 legacy=0x7fe57003b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5892d7640 10 monclient: shutdown 2024-09-16T10:29:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5892d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 >> v1:172.21.15.94:6789/0 conn(0x7fe5700438b0 legacy=0x7fe570045d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5837fe640 1 -- 172.21.15.94:0/4097394300 reap_dead start 2024-09-16T10:29:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5837fe640 1 -- 172.21.15.94:0/4097394300 reap_dead start 2024-09-16T10:29:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 shutdown_connections 2024-09-16T10:29:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 >> 172.21.15.94:0/4097394300 conn(0x7fe5840a59d0 msgr2=0x7fe5840a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 shutdown_connections 2024-09-16T10:29:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.549+0000 7fe5892d7640 1 -- 172.21.15.94:0/4097394300 wait complete. 2024-09-16T10:29:34.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.5 2024-09-16T10:29:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693052722 0 0) 0x7f07480a3250 con 0x7f07480a43c0 2024-09-16T10:29:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986295681 0 0) 0x7f9d480715e0 con 0x7f9d480a1080 2024-09-16T10:29:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f9d3f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f9d3f7fe640 10 monclient(hunting): my global_id is 4236 2024-09-16T10:29:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550649708 0 0) 0x7fe2980a0600 con 0x7fe2980a3860 2024-09-16T10:29:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d30003520 con 0x7f9d480a1080 2024-09-16T10:29:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7fe295ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7fe295ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7fe295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7fe295ffb640 10 monclient(hunting): my global_id is 4245 2024-09-16T10:29:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe280003520 con 0x7fe2980a3860 2024-09-16T10:29:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169312590 0 0) 0x7f71bc070120 con 0x7f71bc09fe50 2024-09-16T10:29:34.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f71bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f71bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f71bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.813+0000 7f71bb7fe640 10 monclient(hunting): my global_id is 4254 2024-09-16T10:29:34.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ac003520 con 0x7f71bc09fe50 2024-09-16T10:29:34.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): my global_id is 4227 2024-09-16T10:29:34.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0730003520 con 0x7f07480a43c0 2024-09-16T10:29:34.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171732583 0 0) 0x7f73ec0a04d0 con 0x7f73ec06c670 2024-09-16T10:29:34.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): my global_id is 4277 2024-09-16T10:29:34.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d8003520 con 0x7f73ec06c670 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861802607 0 0) 0x7f71bc061ae0 con 0x7f71bc0a2090 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 10 monclient(hunting): my global_id is 4286 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc070120 con 0x7f71bc0a2090 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658545569 0 0) 0x7f9d48061ae0 con 0x7f9d480a0590 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 10 monclient(hunting): my global_id is 4268 2024-09-16T10:29:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d480715e0 con 0x7f9d480a0590 2024-09-16T10:29:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677066531 0 0) 0x7f07480a04d0 con 0x7f074806c670 2024-09-16T10:29:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): my global_id is 4270 2024-09-16T10:29:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07480a3250 con 0x7f074806c670 2024-09-16T10:29:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717412671 0 0) 0x7f0748061ae0 con 0x7f07480a38d0 2024-09-16T10:29:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 10 monclient(hunting): my global_id is 4350 2024-09-16T10:29:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07480a04d0 con 0x7f07480a38d0 2024-09-16T10:29:34.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329985363 0 0) 0x7fe298061ae0 con 0x7fe29806c1d0 2024-09-16T10:29:34.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 10 monclient(hunting): my global_id is 4359 2024-09-16T10:29:34.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe280003840 con 0x7fe29806c1d0 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991266528 0 0) 0x7f73ec061ae0 con 0x7f73ec0a38d0 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): my global_id is 4240 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73ec0a04d0 con 0x7f73ec0a38d0 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017348232 0 0) 0x7f73ec0a3250 con 0x7f73ec0a43c0 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 10 monclient(hunting): my global_id is 4298 2024-09-16T10:29:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73ec061ae0 con 0x7f73ec0a43c0 2024-09-16T10:29:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834466520 0 0) 0x7f71bc06abb0 con 0x7f71bc0a15a0 2024-09-16T10:29:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802370826 0 0) 0x7fe2980a0b30 con 0x7fe2980a4350 2024-09-16T10:29:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 10 monclient(hunting): my global_id is 4318 2024-09-16T10:29:34.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe298061ae0 con 0x7fe2980a4350 2024-09-16T10:29:34.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 10 monclient(hunting): my global_id is 4327 2024-09-16T10:29:34.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc061ae0 con 0x7f71bc0a15a0 2024-09-16T10:29:34.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270579239 0 0) 0x7f9d48070910 con 0x7f9d48071860 2024-09-16T10:29:34.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 10 monclient(hunting): my global_id is 4307 2024-09-16T10:29:34.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.814+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d48061ae0 con 0x7f9d48071860 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446022792 0 0) 0x7f71bc061ae0 con 0x7f71bc0a15a0 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 >> v1:172.21.15.94:6792/0 conn(0x7f71bc09fe50 legacy=0x7f71bc139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 >> v1:172.21.15.94:6790/0 conn(0x7f71bc0a2090 legacy=0x7f71bc13cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 1 monclient: found mon.noname-i 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc070650 con 0x7f71bc0a15a0 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f71b0001a50 con 0x7f71bc0a15a0 2024-09-16T10:29:34.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71c2fed640 5 monclient: authenticate success, global_id 4327 2024-09-16T10:29:34.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71c2fed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:34.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3522362283 0 0) 0x7fe298061ae0 con 0x7fe2980a4350 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: dump: 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.830 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.831 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.831 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.831 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.831 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.831 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.831 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71c2fed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 >> v1:172.21.15.94:6792/0 conn(0x7fe2980a3860 legacy=0x7fe2980a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 >> v1:172.21.15.152:6790/0 conn(0x7fe29806c1d0 legacy=0x7fe29813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 1 monclient: found mon.noname-i 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2980a0e10 con 0x7fe2980a4350 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe290001e10 con 0x7fe2980a4350 2024-09-16T10:29:34.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 <== mon.7 v1:172.21.15.152:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71b0001d60 con 0x7f71bc0a15a0 2024-09-16T10:29:34.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe29d6a3640 5 monclient: authenticate success, global_id 4318 2024-09-16T10:29:34.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe29d6a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 1 -- 172.21.15.94:0/4126769015 <== mon.7 v1:172.21.15.152:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f71b0003950 con 0x7f71bc0a15a0 2024-09-16T10:29:34.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71c2fed640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:34.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71c2fed640 4 set_mon_vals no callback set 2024-09-16T10:29:34.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:34.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:34.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7f71bb7fe640 10 monclient: dump: 2024-09-16T10:29:34.834 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.834 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.834 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.835 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: dump: 2024-09-16T10:29:34.836 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.836 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.836 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.837 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe29d6a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 <== mon.7 v1:172.21.15.152:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe290003990 con 0x7fe2980a4350 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe29d6a3640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe29d6a3640 4 set_mon_vals no callback set 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 1 -- 172.21.15.94:0/3705355088 <== mon.7 v1:172.21.15.152:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe290003e80 con 0x7fe2980a4350 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.815+0000 7fe295ffb640 10 monclient: dump: 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.838 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.839 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7f71c2fed640 10 monclient: shutdown 2024-09-16T10:29:34.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7f71c2fed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7f71c2fed640 1 -- 172.21.15.94:0/4126769015 >> v1:172.21.15.152:6792/0 conn(0x7f71bc0a15a0 legacy=0x7f71bc0a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7fe29d6a3640 10 monclient: shutdown 2024-09-16T10:29:34.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7fe29d6a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7fe29d6a3640 1 -- 172.21.15.94:0/3705355088 >> v1:172.21.15.152:6792/0 conn(0x7fe2980a4350 legacy=0x7fe29813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7f71c2fed640 1 -- 172.21.15.94:0/4126769015 shutdown_connections 2024-09-16T10:29:34.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7f71c2fed640 1 -- 172.21.15.94:0/4126769015 >> 172.21.15.94:0/4126769015 conn(0x7f71bc0a6460 msgr2=0x7f71bc0a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7fe29d6a3640 1 -- 172.21.15.94:0/3705355088 shutdown_connections 2024-09-16T10:29:34.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7f71c2fed640 1 -- 172.21.15.94:0/4126769015 shutdown_connections 2024-09-16T10:29:34.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7fe29d6a3640 1 -- 172.21.15.94:0/3705355088 >> 172.21.15.94:0/3705355088 conn(0x7fe2980a5960 msgr2=0x7fe2980a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7f71c2fed640 1 -- 172.21.15.94:0/4126769015 wait complete. 2024-09-16T10:29:34.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7fe29d6a3640 1 -- 172.21.15.94:0/3705355088 shutdown_connections 2024-09-16T10:29:34.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7fe29d6a3640 1 -- 172.21.15.94:0/3705355088 wait complete. 2024-09-16T10:29:34.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7f71c2fed640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7fe29d6a3640 10 monclient: build_initial_monmap 2024-09-16T10:29:34.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7f71c2fed640 10 monclient: monmap: 2024-09-16T10:29:34.843 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.844 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.816+0000 7fe29d6a3640 10 monclient: monmap: 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:34.846 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:34.846 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:34.846 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:34.846 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:34.846 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:34.846 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:34.846 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:34.846 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:34.846 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 1 Processor -- start 2024-09-16T10:29:34.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 1 -- start start 2024-09-16T10:29:34.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 10 monclient: init 2024-09-16T10:29:34.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 1 Processor -- start 2024-09-16T10:29:34.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 1 -- start start 2024-09-16T10:29:34.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 10 monclient: init 2024-09-16T10:29:34.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 10 monclient: _add_conns ranks=[3,2,7,8,0,5,4,6,1] 2024-09-16T10:29:34.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 10 monclient(hunting): picked mon.noname-g con 0x7fe29806c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:34.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 10 monclient(hunting): picked mon.noname-b con 0x7fe2980a3860 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:34.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 10 monclient(hunting): picked mon.noname-i con 0x7fe2980a4350 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:34.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 10 monclient: _add_conns ranks=[2,1,5,3,6,4,7,8,0] 2024-09-16T10:29:34.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29809f750 con 0x7fe2980a3860 2024-09-16T10:29:34.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29809f200 con 0x7fe29806c1d0 2024-09-16T10:29:34.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2981abcd0 con 0x7fe2980a4350 2024-09-16T10:29:34.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 10 monclient(hunting): picked mon.noname-b con 0x7f71bc09fe50 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:34.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe29d6a3640 10 monclient(hunting): authenticate will time out at 742.157874s 2024-09-16T10:29:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 10 monclient(hunting): picked mon.noname-f con 0x7f71bc0a15a0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 10 monclient(hunting): picked mon.noname-h con 0x7f71bc0a2090 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc0a51a0 con 0x7f71bc0a15a0 2024-09-16T10:29:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc127060 con 0x7f71bc09fe50 2024-09-16T10:29:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7f71c2fed640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc1260d0 con 0x7f71bc0a2090 2024-09-16T10:29:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.818+0000 7f71c2fed640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.817+0000 7fe2967fc640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fe2980a3860 0x7fe2981cfb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53466/0 (socket says 172.21.15.94:53466) 2024-09-16T10:29:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.818+0000 7fe2967fc640 1 -- 172.21.15.94:0/199338911 learned_addr learned my addr 172.21.15.94:0/199338911 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.818+0000 7f71c2fed640 10 monclient(hunting): authenticate will time out at 742.158011s 2024-09-16T10:29:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.818+0000 7f71c0d89640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f71bc09fe50 0x7f71bc134c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53472/0 (socket says 172.21.15.94:53472) 2024-09-16T10:29:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.818+0000 7f71c0d89640 1 -- 172.21.15.94:0/4159115209 learned_addr learned my addr 172.21.15.94:0/4159115209 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe2967fc640 1 -- 172.21.15.94:0/199338911 >> v1:172.21.15.94:6790/0 conn(0x7fe2980a3860 legacy=0x7fe2981cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7f71c0d89640 1 -- 172.21.15.94:0/4159115209 >> v1:172.21.15.94:6790/0 conn(0x7f71bc09fe50 legacy=0x7f71bc134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe296ffd640 1 -- 172.21.15.94:0/199338911 >> v1:172.21.15.152:6790/0 conn(0x7fe29806c1d0 legacy=0x7fe2981be5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe2977fe640 1 -- 172.21.15.94:0/199338911 >> v1:172.21.15.152:6792/0 conn(0x7fe2980a4350 legacy=0x7fe2981d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7f71bbfff640 1 -- 172.21.15.94:0/4159115209 >> v1:172.21.15.152:6789/0 conn(0x7f71bc0a15a0 legacy=0x7f71bc1cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7f71c158a640 1 -- 172.21.15.94:0/4159115209 >> v1:172.21.15.152:6791/0 conn(0x7f71bc0a2090 legacy=0x7f71bc1d3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92161013 0 0) 0x7fe2981abcd0 con 0x7fe2980a4350 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient(hunting): my global_id is 4336 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe284003520 con 0x7fe2980a4350 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 417818049 0 0) 0x7fe284003520 con 0x7fe2980a4350 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2981abcd0 con 0x7fe2980a4350 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe290001dd0 con 0x7fe2980a4350 2024-09-16T10:29:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient(hunting): dump: 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3059676161 0 0) 0x7fe2981abcd0 con 0x7fe2980a4350 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 >> v1:172.21.15.152:6790/0 conn(0x7fe29806c1d0 legacy=0x7fe2981be5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 >> v1:172.21.15.94:6790/0 conn(0x7fe2980a3860 legacy=0x7fe2981cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 monclient: found mon.i 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29806bb70 con 0x7fe2980a4350 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe27f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.820+0000 7fe29d6a3640 5 monclient: authenticate success, global_id 4336 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 10 monclient: _renew_subs 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2981abfb0 con 0x7fe2980a4350 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 10 monclient: _renew_subs 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2981aa570 con 0x7fe2980a4350 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe290002020 con 0x7fe2980a4350 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe27f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe290004740 con 0x7fe2980a4350 2024-09-16T10:29:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe2957fa640 4 set_mon_vals no callback set 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe27f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe27f7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe27f7fe640 10 monclient: dump: 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:34.860 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fe290010a40 con 0x7fe2980a4350 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.821+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe278005040 con 0x7fe2980a4350 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.822+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7fe290046420 con 0x7fe2980a4350 2024-09-16T10:29:34.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.822+0000 7fe296ffd640 10 monclient: get_auth_request con 0x7fe284039010 auth_method 0 2024-09-16T10:29:34.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.826+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fe290015960 con 0x7fe2980a4350 2024-09-16T10:29:34.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.826+0000 7fe27f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:34.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.826+0000 7fe27f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:34.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.988+0000 7fe29d6a3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:29:34.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.988+0000 7fe29d6a3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:29:34.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.988+0000 7fe29d6a3640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:34.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.988+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe2780052f0 con 0x7fe2980a4350 2024-09-16T10:29:34.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.988+0000 7fe27f7fe640 1 -- 172.21.15.94:0/199338911 <== mon.7 v1:172.21.15.152:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 1362196188) 0x7fe290013dd0 con 0x7fe2980a4350 2024-09-16T10:29:34.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.988+0000 7fe27f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:29:34.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.988+0000 7fe27f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:34.991 INFO:teuthology.orchestra.run.smithi094.stdout:51539607553 2024-09-16T10:29:34.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.990+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 >> v1:172.21.15.152:6800/2686360781 conn(0x7fe284039010 legacy=0x7fe28403b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.991+0000 7fe29d6a3640 10 monclient: shutdown 2024-09-16T10:29:34.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.991+0000 7fe29d6a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:34.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.991+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 >> v1:172.21.15.152:6792/0 conn(0x7fe2980a4350 legacy=0x7fe2981d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:34.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.991+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 shutdown_connections 2024-09-16T10:29:34.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.991+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 >> 172.21.15.94:0/199338911 conn(0x7fe2980a5960 msgr2=0x7fe2980a75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:34.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.991+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 shutdown_connections 2024-09-16T10:29:34.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:34.991+0000 7fe29d6a3640 1 -- 172.21.15.94:0/199338911 wait complete. 2024-09-16T10:29:35.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3262539862 0 0) 0x7f07480a3250 con 0x7f074806c670 2024-09-16T10:29:35.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 >> v1:172.21.15.94:6792/0 conn(0x7f07480a43c0 legacy=0x7f074813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 >> v1:172.21.15.152:6790/0 conn(0x7f07480a38d0 legacy=0x7f07480a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f07467fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f07467fc640 1 monclient: found mon.noname-c 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f07467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f07467fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07480a0a00 con 0x7f074806c670 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f073c001a60 con 0x7f074806c670 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 10 monclient: dump: 2024-09-16T10:29:35.003 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.004 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.005 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f074db18640 5 monclient: authenticate success, global_id 4270 2024-09-16T10:29:35.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f074db18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:35.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2800075701 0 0) 0x7f9d30003520 con 0x7f9d480a1080 2024-09-16T10:29:35.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 >> v1:172.21.15.152:6791/0 conn(0x7f9d48071860 legacy=0x7f9d4813b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 >> v1:172.21.15.94:6790/0 conn(0x7f9d480a0590 legacy=0x7f9d480a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:35.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 1 monclient: found mon.noname-d 2024-09-16T10:29:35.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d48070d50 con 0x7f9d480a1080 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9d40001a60 con 0x7f9d480a1080 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.000+0000 7f9d3f7fe640 10 monclient: dump: 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d4ee09640 5 monclient: authenticate success, global_id 4236 2024-09-16T10:29:35.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d4ee09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:35.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925711404 0 0) 0x7f71bc1260d0 con 0x7f71bc0a2090 2024-09-16T10:29:35.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): my global_id is 4316 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a0003520 con 0x7f71bc0a2090 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870525387 0 0) 0x7f71bc0a51a0 con 0x7f71bc0a15a0 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): my global_id is 4249 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc1260d0 con 0x7f71bc0a15a0 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620954545 0 0) 0x7f71bc127060 con 0x7f71bc09fe50 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): my global_id is 4295 2024-09-16T10:29:35.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc0a51a0 con 0x7f71bc09fe50 2024-09-16T10:29:35.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 270237948 0 0) 0x7f73ec0a04d0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 >> v1:172.21.15.152:6791/0 conn(0x7f73ec0a43c0 legacy=0x7f73ec13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 >> v1:172.21.15.94:6790/0 conn(0x7f73ec06c670 legacy=0x7f73ec13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: _finish_hunting 0 2024-09-16T10:29:35.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 1 monclient: found mon.noname-f 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.152:6789/0 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec0a0a00 con 0x7f73ec0a38d0 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: _finish_auth 0 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: _check_auth_tickets 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f73e8001ab0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: dump: 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.011 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f422b640 5 monclient: authenticate success, global_id 4240 2024-09-16T10:29:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f422b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3783112719 0 0) 0x7f71a0003520 con 0x7f71bc0a2090 2024-09-16T10:29:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73e8003610 con 0x7f73ec0a38d0 2024-09-16T10:29:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 1 -- 172.21.15.94:0/2796412556 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f73e8003ac0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f0fc5640 10 monclient: dump: 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.015 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.015 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.015 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.015 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.015 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.015 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f422b640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:35.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f73f422b640 4 set_mon_vals no callback set 2024-09-16T10:29:35.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71a0004cf0 con 0x7f71bc0a2090 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f71b4002030 con 0x7f71bc0a2090 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): dump: 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d40001d70 con 0x7f9d480a1080 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d3f7fe640 1 -- 172.21.15.94:0/3206866105 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9d40004be0 con 0x7f9d480a1080 2024-09-16T10:29:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d3f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d3f7fe640 10 monclient: dump: 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d4ee09640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f9d4ee09640 4 set_mon_vals no callback set 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f073c0048a0 con 0x7f074806c670 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 1 -- 172.21.15.94:0/3767715893 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f073c004d50 con 0x7f074806c670 2024-09-16T10:29:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f07467fc640 10 monclient: dump: 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.021 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.021 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.021 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.021 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f074db18640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:35.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f074db18640 4 set_mon_vals no callback set 2024-09-16T10:29:35.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3384406164 0 0) 0x7f71bc0a51a0 con 0x7f71bc09fe50 2024-09-16T10:29:35.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71a0003520 con 0x7f71bc09fe50 2024-09-16T10:29:35.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f71b0003950 con 0x7f71bc09fe50 2024-09-16T10:29:35.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:35.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient(hunting): dump: 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.023 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2994518976 0 0) 0x7f71a0004cf0 con 0x7f71bc0a2090 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 >> v1:172.21.15.94:6790/0 conn(0x7f71bc09fe50 legacy=0x7f71bc134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 >> v1:172.21.15.152:6789/0 conn(0x7f71bc0a15a0 legacy=0x7f71bc1cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 monclient: found mon.h 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.152:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc125e90 con 0x7f71bc0a2090 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.001+0000 7f71b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71b4003c40 con 0x7f71bc0a2090 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71b97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f71b40040f0 con 0x7f71bc0a2090 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71b97fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71b97fa640 10 monclient: dump: 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.026 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71baffd640 4 set_mon_vals no callback set 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71c2fed640 5 monclient: authenticate success, global_id 4316 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71c2fed640 10 monclient: _renew_subs 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71c2fed640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.152:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc126d80 con 0x7f71bc0a2090 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f73f422b640 10 monclient: shutdown 2024-09-16T10:29:35.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71c2fed640 10 monclient: _renew_subs 2024-09-16T10:29:35.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71c2fed640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:35.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.152:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc06ee30 con 0x7f71bc0a2090 2024-09-16T10:29:35.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f73f422b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:35.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f9d4ee09640 10 monclient: shutdown 2024-09-16T10:29:35.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f9d4ee09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:35.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f9d4ee09640 1 -- 172.21.15.94:0/3206866105 >> v1:172.21.15.94:6792/0 conn(0x7f9d480a1080 legacy=0x7f9d4813efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f73f422b640 1 -- 172.21.15.94:0/2796412556 >> v1:172.21.15.152:6789/0 conn(0x7f73ec0a38d0 legacy=0x7f73ec0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f9d4ee09640 1 -- 172.21.15.94:0/3206866105 shutdown_connections 2024-09-16T10:29:35.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f9d4ee09640 1 -- 172.21.15.94:0/3206866105 >> 172.21.15.94:0/3206866105 conn(0x7f9d480a1680 msgr2=0x7f9d480a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:35.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f73f422b640 1 -- 172.21.15.94:0/2796412556 shutdown_connections 2024-09-16T10:29:35.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f73f422b640 1 -- 172.21.15.94:0/2796412556 >> 172.21.15.94:0/2796412556 conn(0x7f73ec0a59d0 msgr2=0x7f73ec0a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:35.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f074db18640 10 monclient: shutdown 2024-09-16T10:29:35.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f074db18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:35.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f074db18640 1 -- 172.21.15.94:0/3767715893 >> v1:172.21.15.94:6791/0 conn(0x7f074806c670 legacy=0x7f074813bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f73f422b640 1 -- 172.21.15.94:0/2796412556 shutdown_connections 2024-09-16T10:29:35.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f73f422b640 1 -- 172.21.15.94:0/2796412556 wait complete. 2024-09-16T10:29:35.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f9d4ee09640 1 -- 172.21.15.94:0/3206866105 shutdown_connections 2024-09-16T10:29:35.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f9d4ee09640 1 -- 172.21.15.94:0/3206866105 wait complete. 2024-09-16T10:29:35.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f074db18640 1 -- 172.21.15.94:0/3767715893 shutdown_connections 2024-09-16T10:29:35.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.002+0000 7f074db18640 1 -- 172.21.15.94:0/3767715893 >> 172.21.15.94:0/3767715893 conn(0x7f07480a59d0 msgr2=0x7f07480a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:35.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f074db18640 1 -- 172.21.15.94:0/3767715893 shutdown_connections 2024-09-16T10:29:35.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f074db18640 1 -- 172.21.15.94:0/3767715893 wait complete. 2024-09-16T10:29:35.032 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607553 for osd.2 2024-09-16T10:29:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f71b4004290 con 0x7f71bc0a2090 2024-09-16T10:29:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f73f422b640 10 monclient: build_initial_monmap 2024-09-16T10:29:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f73f422b640 10 monclient: monmap: 2024-09-16T10:29:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f9d4ee09640 10 monclient: build_initial_monmap 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f9d4ee09640 10 monclient: monmap: 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:35.035 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:35.036 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:35.036 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:35.036 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:35.036 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f074db18640 10 monclient: build_initial_monmap 2024-09-16T10:29:35.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f71c2fed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f074db18640 10 monclient: monmap: 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:35.037 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:35.038 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:35.038 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:35.038 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f71c2fed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f71c2fed640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:35.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f71b40461b0 con 0x7f71bc0a2090 2024-09-16T10:29:35.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.003+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.152:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71bc1272a0 con 0x7f71bc0a2090 2024-09-16T10:29:35.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f71c0d89640 10 monclient: get_auth_request con 0x7f71a0039a90 auth_method 0 2024-09-16T10:29:35.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 1 Processor -- start 2024-09-16T10:29:35.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f9d4ee09640 1 Processor -- start 2024-09-16T10:29:35.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 1 -- start start 2024-09-16T10:29:35.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 10 monclient: init 2024-09-16T10:29:35.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f9d4ee09640 1 -- start start 2024-09-16T10:29:35.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f9d4ee09640 10 monclient: init 2024-09-16T10:29:35.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f074db18640 1 Processor -- start 2024-09-16T10:29:35.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f074db18640 1 -- start start 2024-09-16T10:29:35.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f074db18640 10 monclient: init 2024-09-16T10:29:35.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:35.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 10 monclient: _add_conns ranks=[6,0,1,5,2,4,3,8,7] 2024-09-16T10:29:35.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 10 monclient(hunting): picked mon.noname-d con 0x7f73ec06c670 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:35.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 10 monclient(hunting): picked mon.noname-a con 0x7f73ec0a38d0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:35.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 10 monclient(hunting): picked mon.noname-f con 0x7f73ec0a43c0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:35.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec120520 con 0x7f73ec0a38d0 2024-09-16T10:29:35.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec13b3b0 con 0x7f73ec0a43c0 2024-09-16T10:29:35.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec13fe60 con 0x7f73ec06c670 2024-09-16T10:29:35.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:35.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f422b640 10 monclient(hunting): authenticate will time out at 742.344764s 2024-09-16T10:29:35.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f17c6640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f73ec0a38d0 0x7f73ec1d41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59424/0 (socket says 172.21.15.94:59424) 2024-09-16T10:29:35.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.004+0000 7f73f17c6640 1 -- 172.21.15.94:0/1711867845 learned_addr learned my addr 172.21.15.94:0/1711867845 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:35.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:35.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 10 monclient: _add_conns ranks=[6,8,4,1,0,3,5,7,2] 2024-09-16T10:29:35.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 10 monclient(hunting): picked mon.noname-d con 0x7f9d48071860 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:35.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:35.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 10 monclient: _add_conns ranks=[7,6,4,5,8,2,0,3,1] 2024-09-16T10:29:35.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 10 monclient(hunting): picked mon.noname-e con 0x7f9d480a0590 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:35.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 10 monclient(hunting): picked mon.noname-i con 0x7f074806c670 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:35.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 10 monclient(hunting): picked mon.noname-c con 0x7f9d480a1080 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:35.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d3ffff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f9d480a0590 0x7f9d481cfaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:58282/0 (socket says 172.21.15.94:58282) 2024-09-16T10:29:35.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4809d830 con 0x7f9d480a1080 2024-09-16T10:29:35.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d3ffff640 1 -- 172.21.15.94:0/4267689082 learned_addr learned my addr 172.21.15.94:0/4267689082 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:35.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 10 monclient(hunting): picked mon.noname-d con 0x7f07480a38d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:35.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 10 monclient(hunting): picked mon.noname-c con 0x7f07480a43c0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:35.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074809f240 con 0x7f07480a43c0 2024-09-16T10:29:35.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4809d2e0 con 0x7f9d48071860 2024-09-16T10:29:35.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0748126e00 con 0x7f07480a38d0 2024-09-16T10:29:35.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07481265a0 con 0x7f074806c670 2024-09-16T10:29:35.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f07477fe640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7f074806c670 0x7f07481366f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.94:35850/0 (socket says 172.21.15.94:35850) 2024-09-16T10:29:35.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:35.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f07477fe640 1 -- 172.21.15.94:0/1346191961 learned_addr learned my addr 172.21.15.94:0/1346191961 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:35.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f074db18640 10 monclient(hunting): authenticate will time out at 742.345458s 2024-09-16T10:29:35.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d480a01a0 con 0x7f9d480a0590 2024-09-16T10:29:35.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:35.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.005+0000 7f9d4ee09640 10 monclient(hunting): authenticate will time out at 742.345489s 2024-09-16T10:29:35.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.006+0000 7f73f17c6640 1 -- 172.21.15.94:0/1711867845 >> v1:172.21.15.94:6789/0 conn(0x7f73ec0a38d0 legacy=0x7f73ec1d41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.006+0000 7f73f1fc7640 1 -- 172.21.15.94:0/1711867845 >> v1:172.21.15.94:6792/0 conn(0x7f73ec06c670 legacy=0x7f73ec13aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73f27c8640 1 -- 172.21.15.94:0/1711867845 >> v1:172.21.15.152:6789/0 conn(0x7f73ec0a43c0 legacy=0x7f73ec1d7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625680603 0 0) 0x7f73d8003520 con 0x7f73ec0a38d0 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): my global_id is 4320 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d0003520 con 0x7f73ec0a38d0 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997331993 0 0) 0x7f73ec13b3b0 con 0x7f73ec0a43c0 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): my global_id is 4258 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d8003520 con 0x7f73ec0a43c0 2024-09-16T10:29:35.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096989591 0 0) 0x7f73d0003520 con 0x7f73ec0a38d0 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73ec13b3b0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f73dc0047e0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): dump: 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.049 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.050 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.050 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.050 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.050 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.050 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.050 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.050 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:35.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f9d3ffff640 1 -- 172.21.15.94:0/4267689082 >> v1:172.21.15.94:6793/0 conn(0x7f9d480a0590 legacy=0x7f9d481cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2986208757 0 0) 0x7f73d8003520 con 0x7f73ec0a43c0 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73d0003520 con 0x7f73ec0a43c0 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f73e4003cd0 con 0x7f73ec0a43c0 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient(hunting): dump: 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.051 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f9d4d3a6640 1 -- 172.21.15.94:0/4267689082 >> v1:172.21.15.94:6791/0 conn(0x7f9d480a1080 legacy=0x7f9d481d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3676771568 0 0) 0x7f73ec13b3b0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f07477fe640 1 -- 172.21.15.94:0/1346191961 >> v1:172.21.15.152:6792/0 conn(0x7f074806c670 legacy=0x7f07481366f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 >> v1:172.21.15.94:6792/0 conn(0x7f73ec06c670 legacy=0x7f73ec13aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 >> v1:172.21.15.152:6789/0 conn(0x7f73ec0a43c0 legacy=0x7f73ec1d7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:35.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 monclient: found mon.a 2024-09-16T10:29:35.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238064926 0 0) 0x7f9d480a01a0 con 0x7f9d480a0590 2024-09-16T10:29:35.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f9d3d7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f9d3d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f9d3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:35.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:35.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec06baf0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:35.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:35.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:35.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73f422b640 5 monclient: authenticate success, global_id 4320 2024-09-16T10:29:35.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f9d3d7fa640 10 monclient(hunting): my global_id is 4301 2024-09-16T10:29:35.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d34003520 con 0x7f9d480a0590 2024-09-16T10:29:35.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73f422b640 10 monclient: _renew_subs 2024-09-16T10:29:35.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73f422b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:35.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ec06b5a0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333239706 0 0) 0x7f07481265a0 con 0x7f074806c670 2024-09-16T10:29:35.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f072ffff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f072ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f072ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f072ffff640 10 monclient(hunting): my global_id is 4345 2024-09-16T10:29:35.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0734003520 con 0x7f074806c670 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73f422b640 10 monclient: _renew_subs 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73f422b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ec09fac0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73dc004d80 con 0x7f73ec0a38d0 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e3fff640 4 set_mon_vals no callback set 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f73dc0052b0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:35.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.007+0000 7f73e27fc640 10 monclient: dump: 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.061 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.062 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.062 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.062 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.062 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3624031412 0 0) 0x7f0734003520 con 0x7f074806c670 2024-09-16T10:29:35.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1661830615 0 0) 0x7f9d34003520 con 0x7f9d480a0590 2024-09-16T10:29:35.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07481265a0 con 0x7f074806c670 2024-09-16T10:29:35.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0738001d10 con 0x7f074806c670 2024-09-16T10:29:35.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:35.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient(hunting): dump: 2024-09-16T10:29:35.063 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.065 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.065 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.065 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d480a01a0 con 0x7f9d480a0590 2024-09-16T10:29:35.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9d38003610 con 0x7f9d480a0590 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 10 monclient(hunting): dump: 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.066 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:35.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4035680183 0 0) 0x7f07481265a0 con 0x7f074806c670 2024-09-16T10:29:35.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 >> v1:172.21.15.94:6792/0 conn(0x7f07480a38d0 legacy=0x7f07481cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 >> v1:172.21.15.94:6791/0 conn(0x7f07480a43c0 legacy=0x7f07481d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 monclient: found mon.i 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0748125460 con 0x7f074806c670 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient: _finish_auth 0 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f074db18640 5 monclient: authenticate success, global_id 4345 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f074db18640 10 monclient: _renew_subs 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f074db18640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0748126b20 con 0x7f074806c670 2024-09-16T10:29:35.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f074db18640 10 monclient: _renew_subs 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f074db18640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0748136fe0 con 0x7f074806c670 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0738003b80 con 0x7f074806c670 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0738003750 con 0x7f074806c670 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f0745ffb640 4 set_mon_vals no callback set 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f072ffff640 10 monclient: dump: 2024-09-16T10:29:35.070 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.071 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.072 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.072 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2080900065 0 0) 0x7f9d480a01a0 con 0x7f9d480a0590 2024-09-16T10:29:35.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 >> v1:172.21.15.94:6792/0 conn(0x7f9d48071860 legacy=0x7f9d48132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 >> v1:172.21.15.94:6791/0 conn(0x7f9d480a1080 legacy=0x7f9d481d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:35.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 1 monclient: found mon.e 2024-09-16T10:29:35.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:35.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:35.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d48121df0 con 0x7f9d480a0590 2024-09-16T10:29:35.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:35.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f73dc005490 con 0x7f73ec0a38d0 2024-09-16T10:29:35.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:35.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:35.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d4ee09640 5 monclient: authenticate success, global_id 4301 2024-09-16T10:29:35.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d38003cd0 con 0x7f9d480a0590 2024-09-16T10:29:35.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d4ee09640 10 monclient: _renew_subs 2024-09-16T10:29:35.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d4ee09640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:35.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.008+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6793/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d48123360 con 0x7f9d480a0590 2024-09-16T10:29:35.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f9d4ee09640 10 monclient: _renew_subs 2024-09-16T10:29:35.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f9d4ee09640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:35.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d48133080 con 0x7f9d480a0590 2024-09-16T10:29:35.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f71b4015400 con 0x7f71bc0a2090 2024-09-16T10:29:35.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f71b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f71b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:35.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f73f422b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f73f422b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f73f422b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:35.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d4005040 con 0x7f73ec0a38d0 2024-09-16T10:29:35.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f73dc046010 con 0x7f73ec0a38d0 2024-09-16T10:29:35.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f07380040f0 con 0x7f074806c670 2024-09-16T10:29:35.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f074db18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f074db18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f074db18640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:35.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0728005040 con 0x7f074806c670 2024-09-16T10:29:35.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f0738046020 con 0x7f074806c670 2024-09-16T10:29:35.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:35.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9d38004180 con 0x7f9d480a0590 2024-09-16T10:29:35.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d3d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d3d7fa640 10 monclient: dump: 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.078 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.079 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.079 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.079 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.079 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.079 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d3e7fc640 4 set_mon_vals no callback set 2024-09-16T10:29:35.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.009+0000 7f73f1fc7640 10 monclient: get_auth_request con 0x7f73d00396c0 auth_method 0 2024-09-16T10:29:35.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f9d380106c0 con 0x7f9d480a0590 2024-09-16T10:29:35.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f9d38046050 con 0x7f9d480a0590 2024-09-16T10:29:35.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d4ee09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d4ee09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d4ee09640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:35.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.010+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d28005040 con 0x7f9d480a0590 2024-09-16T10:29:35.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.012+0000 7f0746ffd640 10 monclient: get_auth_request con 0x7f0734039070 auth_method 0 2024-09-16T10:29:35.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.012+0000 7f9d4cba5640 10 monclient: get_auth_request con 0x7f9d34039060 auth_method 0 2024-09-16T10:29:35.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.014+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f73dc015510 con 0x7f73ec0a38d0 2024-09-16T10:29:35.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.014+0000 7f73e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.014+0000 7f73e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:35.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.014+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f0738015270 con 0x7f074806c670 2024-09-16T10:29:35.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.014+0000 7f072ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.014+0000 7f072ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:35.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.015+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f9d380153c0 con 0x7f9d480a0590 2024-09-16T10:29:35.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.015+0000 7f9d3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.015+0000 7f9d3d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:35.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.185+0000 7f074db18640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:29:35.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.185+0000 7f074db18640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:29:35.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.185+0000 7f074db18640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:35.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.185+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f07280052f0 con 0x7f074806c670 2024-09-16T10:29:35.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.185+0000 7f072ffff640 1 -- 172.21.15.94:0/1346191961 <== mon.7 v1:172.21.15.152:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 555833778) 0x7f0738013910 con 0x7f074806c670 2024-09-16T10:29:35.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.185+0000 7f072ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:29:35.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.185+0000 7f072ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:35.187 INFO:teuthology.orchestra.run.smithi094.stdout:47244640258 2024-09-16T10:29:35.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 >> v1:172.21.15.152:6800/2686360781 conn(0x7f0734039070 legacy=0x7f073403b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f074db18640 10 monclient: shutdown 2024-09-16T10:29:35.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f074db18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:35.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 >> v1:172.21.15.152:6792/0 conn(0x7f074806c670 legacy=0x7f07481366f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 shutdown_connections 2024-09-16T10:29:35.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 >> 172.21.15.94:0/1346191961 conn(0x7f07480a59d0 msgr2=0x7f074809ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:35.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f73f422b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:29:35.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f73f422b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:29:35.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f73f422b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:35.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f73d40052f0 con 0x7f73ec0a38d0 2024-09-16T10:29:35.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.187+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 shutdown_connections 2024-09-16T10:29:35.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.188+0000 7f074db18640 1 -- 172.21.15.94:0/1346191961 wait complete. 2024-09-16T10:29:35.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.188+0000 7f73e27fc640 1 -- 172.21.15.94:0/1711867845 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 555833778) 0x7f73dc013980 con 0x7f73ec0a38d0 2024-09-16T10:29:35.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.188+0000 7f73e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:29:35.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.188+0000 7f73e27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:35.191 INFO:teuthology.orchestra.run.smithi094.stdout:47244640258 2024-09-16T10:29:35.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.190+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 >> v1:172.21.15.152:6800/2686360781 conn(0x7f73d00396c0 legacy=0x7f73d003bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.190+0000 7f73f422b640 10 monclient: shutdown 2024-09-16T10:29:35.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.190+0000 7f73f422b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:35.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.190+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 >> v1:172.21.15.94:6789/0 conn(0x7f73ec0a38d0 legacy=0x7f73ec1d41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.190+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 shutdown_connections 2024-09-16T10:29:35.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.190+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 >> 172.21.15.94:0/1711867845 conn(0x7f73ec0a59d0 msgr2=0x7f73ec0a6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:35.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.190+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 shutdown_connections 2024-09-16T10:29:35.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.190+0000 7f73f422b640 1 -- 172.21.15.94:0/1711867845 wait complete. 2024-09-16T10:29:35.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2024-09-16T10:29:35.207 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-09-16T10:29:35.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.206+0000 7f71c2fed640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:29:35.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.206+0000 7f71c2fed640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:29:35.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.206+0000 7f71c2fed640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:35.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.206+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 --> v1:172.21.15.152:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f71bc06e720 con 0x7f71bc0a2090 2024-09-16T10:29:35.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.206+0000 7f71b97fa640 1 -- 172.21.15.94:0/4159115209 <== mon.5 v1:172.21.15.152:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 555833778) 0x7f71b4015220 con 0x7f71bc0a2090 2024-09-16T10:29:35.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.206+0000 7f71b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:29:35.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.206+0000 7f71b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:35.208 INFO:teuthology.orchestra.run.smithi094.stdout:47244640258 2024-09-16T10:29:35.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.209+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 >> v1:172.21.15.152:6800/2686360781 conn(0x7f71a0039a90 legacy=0x7f71a003bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.209+0000 7f71c2fed640 10 monclient: shutdown 2024-09-16T10:29:35.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.209+0000 7f71c2fed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:35.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.209+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 >> v1:172.21.15.152:6791/0 conn(0x7f71bc0a2090 legacy=0x7f71bc1d3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.209+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 shutdown_connections 2024-09-16T10:29:35.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.209+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 >> 172.21.15.94:0/4159115209 conn(0x7f71bc0a6460 msgr2=0x7f71bc0a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:35.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.209+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 shutdown_connections 2024-09-16T10:29:35.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.209+0000 7f71c2fed640 1 -- 172.21.15.94:0/4159115209 wait complete. 2024-09-16T10:29:35.225 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.0 2024-09-16T10:29:35.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.223+0000 7f9d4ee09640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:29:35.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.223+0000 7f9d4ee09640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:29:35.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.223+0000 7f9d4ee09640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:35.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.223+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 --> v1:172.21.15.94:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f9d280052f0 con 0x7f9d480a0590 2024-09-16T10:29:35.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.223+0000 7f9d3d7fa640 1 -- 172.21.15.94:0/4267689082 <== mon.8 v1:172.21.15.94:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 847495621) 0x7f9d380151e0 con 0x7f9d480a0590 2024-09-16T10:29:35.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.223+0000 7f9d3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:29:35.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.223+0000 7f9d3d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:35.226 INFO:teuthology.orchestra.run.smithi094.stdout:47244640259 2024-09-16T10:29:35.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.225+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 >> v1:172.21.15.152:6800/2686360781 conn(0x7f9d34039060 legacy=0x7f9d3403b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.225+0000 7f9d4ee09640 10 monclient: shutdown 2024-09-16T10:29:35.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.225+0000 7f9d4ee09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:35.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.225+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 >> v1:172.21.15.94:6793/0 conn(0x7f9d480a0590 legacy=0x7f9d481cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.225+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 shutdown_connections 2024-09-16T10:29:35.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.225+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 >> 172.21.15.94:0/4267689082 conn(0x7f9d480a1680 msgr2=0x7f9d480a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:35.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.225+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 shutdown_connections 2024-09-16T10:29:35.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.225+0000 7f9d4ee09640 1 -- 172.21.15.94:0/4267689082 wait complete. 2024-09-16T10:29:35.238 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.3 2024-09-16T10:29:35.563 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T10:29:35.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.659+0000 7f11fa5a3640 10 monclient: get_monmap_and_config 2024-09-16T10:29:35.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.659+0000 7f11fa5a3640 10 monclient: build_initial_monmap 2024-09-16T10:29:35.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.659+0000 7f11fa5a3640 10 monclient: monmap: 2024-09-16T10:29:35.661 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:35.661 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.661 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:35.660400+0000 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:35.660400+0000 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:35.662 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 1 Processor -- start 2024-09-16T10:29:35.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 1 -- start start 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 10 monclient: init 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 10 monclient: _add_conns ranks=[4,6,1,8,2,0,7,3,5] 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 10 monclient(hunting): picked mon.noname-c con 0x7f11f41368e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 10 monclient(hunting): picked mon.noname-d con 0x7f11f4136cc0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 10 monclient(hunting): picked mon.noname-f con 0x7f11f41370a0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f4061ae0 con 0x7f11f41370a0 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f40a03e0 con 0x7f11f41368e0 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f406c8d0 con 0x7f11f4136cc0 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11fa5a3640 10 monclient(hunting): authenticate will time out at 743.000828s 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11f37fe640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f11f4136cc0 0x7f11f413b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60278/0 (socket says 172.21.15.94:60278) 2024-09-16T10:29:35.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11f37fe640 1 -- 172.21.15.94:0/3080517059 learned_addr learned my addr 172.21.15.94:0/3080517059 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.660+0000 7f11f3fff640 1 --1- 172.21.15.94:0/3080517059 >> v1:172.21.15.94:6791/0 conn(0x7f11f41368e0 0x7f11f40a1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43392/0 (socket says 172.21.15.94:43392) 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f3fff640 1 -- 172.21.15.94:0/3080517059 >> v1:172.21.15.94:6791/0 conn(0x7f11f41368e0 legacy=0x7f11f40a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f37fe640 1 -- 172.21.15.94:0/3080517059 >> v1:172.21.15.94:6792/0 conn(0x7f11f4136cc0 legacy=0x7f11f413b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094563344 0 0) 0x7f11f40a03e0 con 0x7f11f41368e0 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient(hunting): my global_id is 4288 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc003520 con 0x7f11f41368e0 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088803335 0 0) 0x7f11f406c8d0 con 0x7f11f4136cc0 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient(hunting): my global_id is 4272 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f40a03e0 con 0x7f11f4136cc0 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f8b40640 1 -- 172.21.15.94:0/3080517059 >> v1:172.21.15.152:6789/0 conn(0x7f11f41370a0 legacy=0x7f11f413ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455340274 0 0) 0x7f11f40a03e0 con 0x7f11f4136cc0 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 >> v1:172.21.15.94:6791/0 conn(0x7f11f41368e0 legacy=0x7f11f40a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 >> v1:172.21.15.152:6789/0 conn(0x7f11f41370a0 legacy=0x7f11f413ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 monclient: found mon.noname-d 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f40a0680 con 0x7f11f4136cc0 2024-09-16T10:29:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11e8001990 con 0x7f11f4136cc0 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11fa5a3640 5 monclient: authenticate success, global_id 4272 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11fa5a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: dump: 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11fa5a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11e8001ca0 con 0x7f11f4136cc0 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11fa5a3640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 1 -- 172.21.15.94:0/3080517059 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11e8004be0 con 0x7f11f4136cc0 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11fa5a3640 4 set_mon_vals no callback set 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.663+0000 7f11f2ffd640 10 monclient: dump: 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.664+0000 7f11fa5a3640 10 monclient: shutdown 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.664+0000 7f11fa5a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:35.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.664+0000 7f11fa5a3640 1 -- 172.21.15.94:0/3080517059 >> v1:172.21.15.94:6792/0 conn(0x7f11f4136cc0 legacy=0x7f11f413b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.664+0000 7f11fa5a3640 1 -- 172.21.15.94:0/3080517059 shutdown_connections 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.664+0000 7f11fa5a3640 1 -- 172.21.15.94:0/3080517059 >> 172.21.15.94:0/3080517059 conn(0x7f11f40a58d0 msgr2=0x7f11f40a5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.664+0000 7f11fa5a3640 1 -- 172.21.15.94:0/3080517059 shutdown_connections 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.664+0000 7f11fa5a3640 1 -- 172.21.15.94:0/3080517059 wait complete. 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.664+0000 7f11fa5a3640 10 monclient: build_initial_monmap 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.664+0000 7f11fa5a3640 10 monclient: monmap: 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.676 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 1 Processor -- start 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 1 -- start start 2024-09-16T10:29:35.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 10 monclient: init 2024-09-16T10:29:35.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:35.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 10 monclient: _add_conns ranks=[1,4,5,3,8,7,6,0,2] 2024-09-16T10:29:35.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 10 monclient(hunting): picked mon.noname-f con 0x7f11f41368e0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:35.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 10 monclient(hunting): picked mon.noname-c con 0x7f11f4136cc0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:35.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 10 monclient(hunting): picked mon.noname-h con 0x7f11f41370a0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:35.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f4122730 con 0x7f11f41368e0 2024-09-16T10:29:35.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f4120cb0 con 0x7f11f4136cc0 2024-09-16T10:29:35.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f4120240 con 0x7f11f41370a0 2024-09-16T10:29:35.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:35.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11fa5a3640 10 monclient(hunting): authenticate will time out at 743.005704s 2024-09-16T10:29:35.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11f37fe640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f11f4136cc0 0x7f11f411fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43398/0 (socket says 172.21.15.94:43398) 2024-09-16T10:29:35.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.665+0000 7f11f37fe640 1 -- 172.21.15.94:0/1725086565 learned_addr learned my addr 172.21.15.94:0/1725086565 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:35.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.667+0000 7f11f37fe640 1 -- 172.21.15.94:0/1725086565 >> v1:172.21.15.94:6791/0 conn(0x7f11f4136cc0 legacy=0x7f11f411fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.667+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216194757 0 0) 0x7f11f4120cb0 con 0x7f11f4136cc0 2024-09-16T10:29:35.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): my global_id is 4297 2024-09-16T10:29:35.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e0003520 con 0x7f11f4136cc0 2024-09-16T10:29:35.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f8b40640 1 -- 172.21.15.94:0/1725086565 >> v1:172.21.15.152:6791/0 conn(0x7f11f41370a0 legacy=0x7f11f41d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f3fff640 1 -- 172.21.15.94:0/1725086565 >> v1:172.21.15.152:6789/0 conn(0x7f11f41368e0 legacy=0x7f11f411f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745918375 0 0) 0x7f11e0003520 con 0x7f11f4136cc0 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f4120cb0 con 0x7f11f4136cc0 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11e8004b20 con 0x7f11f4136cc0 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): dump: 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:35.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442297741 0 0) 0x7f11f4120240 con 0x7f11f41370a0 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): my global_id is 4325 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e0003520 con 0x7f11f41370a0 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579390745 0 0) 0x7f11f4122730 con 0x7f11f41368e0 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient(hunting): my global_id is 4276 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f4120240 con 0x7f11f41368e0 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1494718739 0 0) 0x7f11f4120cb0 con 0x7f11f4136cc0 2024-09-16T10:29:35.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 >> v1:172.21.15.152:6791/0 conn(0x7f11f41370a0 legacy=0x7f11f41d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 >> v1:172.21.15.152:6789/0 conn(0x7f11f41368e0 legacy=0x7f11f411f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient: _finish_hunting 0 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 monclient: found mon.c 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f4120f30 con 0x7f11f4136cc0 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient: _finish_auth 0 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11fa5a3640 5 monclient: authenticate success, global_id 4297 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11fa5a3640 10 monclient: _renew_subs 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11fa5a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:35.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f40a8690 con 0x7f11f4136cc0 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11fa5a3640 10 monclient: _renew_subs 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11fa5a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f4121150 con 0x7f11f4136cc0 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11e8001250 con 0x7f11f4136cc0 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11e8005550 con 0x7f11f4136cc0 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f27fc640 4 set_mon_vals no callback set 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.668+0000 7f11f0ff9640 10 monclient: dump: 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:35.689 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.669+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f11e80105b0 con 0x7f11f4136cc0 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.669+0000 7f11fa5a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.669+0000 7f11fa5a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.669+0000 7f11fa5a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.669+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d4005040 con 0x7f11f4136cc0 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.669+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f11e8046140 con 0x7f11f4136cc0 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.669+0000 7f11f3fff640 10 monclient: get_auth_request con 0x7f11e0039350 auth_method 0 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.673+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f11e80156e0 con 0x7f11f4136cc0 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.673+0000 7f11f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:35.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.674+0000 7f11f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:35.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.837+0000 7f11fa5a3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:35.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.837+0000 7f11fa5a3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:35.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.837+0000 7f11fa5a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:35.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.837+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f11d40052f0 con 0x7f11f4136cc0 2024-09-16T10:29:35.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.838+0000 7f11f0ff9640 1 -- 172.21.15.94:0/1725086565 <== mon.4 v1:172.21.15.94:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 555833778) 0x7f11e8013b50 con 0x7f11f4136cc0 2024-09-16T10:29:35.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.838+0000 7f11f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:35.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.838+0000 7f11f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:35.840 INFO:teuthology.orchestra.run.smithi094.stdout:47244640258 2024-09-16T10:29:35.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.840+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 >> v1:172.21.15.152:6800/2686360781 conn(0x7f11e0039350 legacy=0x7f11e003b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.840+0000 7f11fa5a3640 10 monclient: shutdown 2024-09-16T10:29:35.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.840+0000 7f11fa5a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:35.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.840+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 >> v1:172.21.15.94:6791/0 conn(0x7f11f4136cc0 legacy=0x7f11f411fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:35.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.840+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 shutdown_connections 2024-09-16T10:29:35.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.840+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 >> 172.21.15.94:0/1725086565 conn(0x7f11f40a58d0 msgr2=0x7f11f4135ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:35.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.840+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 shutdown_connections 2024-09-16T10:29:35.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:35.840+0000 7f11fa5a3640 1 -- 172.21.15.94:0/1725086565 wait complete. 2024-09-16T10:29:35.852 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.5 2024-09-16T10:29:36.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:29:36.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.129+0000 7f45cc6c9640 10 monclient: get_monmap_and_config 2024-09-16T10:29:36.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.129+0000 7f45cc6c9640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.129+0000 7f45cc6c9640 10 monclient: monmap: 2024-09-16T10:29:36.131 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.131 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:36.130321+0000 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:36.130321+0000 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.132 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 1 Processor -- start 2024-09-16T10:29:36.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 1 -- start start 2024-09-16T10:29:36.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 10 monclient: init 2024-09-16T10:29:36.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 10 monclient: _add_conns ranks=[1,0,6,5,3,2,4,8,7] 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 10 monclient(hunting): picked mon.noname-f con 0x7f45c40a37d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 10 monclient(hunting): picked mon.noname-a con 0x7f45c40a42c0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 10 monclient(hunting): picked mon.noname-d con 0x7f45c40a1740 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c4061ae0 con 0x7f45c40a42c0 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c40a4890 con 0x7f45c40a37d0 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c406c490 con 0x7f45c40a1740 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cc6c9640 10 monclient(hunting): authenticate will time out at 743.470766s 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cac66640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f45c40a1740 0x7f45c413ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60280/0 (socket says 172.21.15.94:60280) 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45cac66640 1 -- 172.21.15.94:0/4257800980 learned_addr learned my addr 172.21.15.94:0/4257800980 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.130+0000 7f45c9c64640 1 --1- 172.21.15.94:0/4257800980 >> v1:172.21.15.94:6789/0 conn(0x7f45c40a42c0 0x7f45c413b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41470/0 (socket says 172.21.15.94:41470) 2024-09-16T10:29:36.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45cac66640 1 -- 172.21.15.94:0/4257800980 >> v1:172.21.15.94:6792/0 conn(0x7f45c40a1740 legacy=0x7f45c413ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9c64640 1 -- 172.21.15.94:0/4257800980 >> v1:172.21.15.94:6789/0 conn(0x7f45c40a42c0 legacy=0x7f45c413b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959980695 0 0) 0x7f45c406c490 con 0x7f45c40a1740 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient(hunting): my global_id is 4281 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b0003520 con 0x7f45c40a1740 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706774147 0 0) 0x7f45c4061ae0 con 0x7f45c40a42c0 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient(hunting): my global_id is 4329 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c406c490 con 0x7f45c40a42c0 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45ca465640 1 -- 172.21.15.94:0/4257800980 >> v1:172.21.15.152:6789/0 conn(0x7f45c40a37d0 legacy=0x7f45c40a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986916878 0 0) 0x7f45b0003520 con 0x7f45c40a1740 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 >> v1:172.21.15.152:6789/0 conn(0x7f45c40a37d0 legacy=0x7f45c40a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 >> v1:172.21.15.94:6789/0 conn(0x7f45c40a42c0 legacy=0x7f45c413b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 monclient: found mon.noname-d 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c40a0d10 con 0x7f45c40a1740 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45cc6c9640 5 monclient: authenticate success, global_id 4281 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f45bc001de0 con 0x7f45c40a1740 2024-09-16T10:29:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45cc6c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: dump: 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.141 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45cc6c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45bc0020f0 con 0x7f45c40a1740 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 1 -- 172.21.15.94:0/4257800980 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f45bc004f60 con 0x7f45c40a1740 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45cc6c9640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45cc6c9640 4 set_mon_vals no callback set 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45c9463640 10 monclient: dump: 2024-09-16T10:29:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45cc6c9640 10 monclient: shutdown 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45cc6c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.133+0000 7f45cc6c9640 1 -- 172.21.15.94:0/4257800980 >> v1:172.21.15.94:6792/0 conn(0x7f45c40a1740 legacy=0x7f45c413ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.134+0000 7f45cc6c9640 1 -- 172.21.15.94:0/4257800980 shutdown_connections 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.134+0000 7f45cc6c9640 1 -- 172.21.15.94:0/4257800980 >> 172.21.15.94:0/4257800980 conn(0x7f45c40a9460 msgr2=0x7f45c40a58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.134+0000 7f45cc6c9640 1 -- 172.21.15.94:0/4257800980 shutdown_connections 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.134+0000 7f45cc6c9640 1 -- 172.21.15.94:0/4257800980 wait complete. 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.134+0000 7f45cc6c9640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.134+0000 7f45cc6c9640 10 monclient: monmap: 2024-09-16T10:29:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.145 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.134+0000 7f45cc6c9640 1 Processor -- start 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 1 -- start start 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 10 monclient: init 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 10 monclient: _add_conns ranks=[0,1,3,7,8,2,5,6,4] 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 10 monclient(hunting): picked mon.noname-a con 0x7f45c40a1740 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 10 monclient(hunting): picked mon.noname-f con 0x7f45c40a37d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 10 monclient(hunting): picked mon.noname-g con 0x7f45c40a42c0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c41abf90 con 0x7f45c40a1740 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c41aaa70 con 0x7f45c40a37d0 2024-09-16T10:29:36.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c40a7b80 con 0x7f45c40a42c0 2024-09-16T10:29:36.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45cc6c9640 10 monclient(hunting): authenticate will time out at 743.475498s 2024-09-16T10:29:36.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45ca465640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f45c40a1740 0x7f45c41be8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41484/0 (socket says 172.21.15.94:41484) 2024-09-16T10:29:36.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.135+0000 7f45ca465640 1 -- 172.21.15.94:0/3379963626 learned_addr learned my addr 172.21.15.94:0/3379963626 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45ca465640 1 -- 172.21.15.94:0/3379963626 >> v1:172.21.15.94:6789/0 conn(0x7f45c40a1740 legacy=0x7f45c41be8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362342442 0 0) 0x7f45c41abf90 con 0x7f45c40a1740 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 10 monclient(hunting): my global_id is 4338 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a8003520 con 0x7f45c40a1740 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2490827296 0 0) 0x7f45a8003520 con 0x7f45c40a1740 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45c9c64640 1 -- 172.21.15.94:0/3379963626 >> v1:172.21.15.152:6789/0 conn(0x7f45c40a37d0 legacy=0x7f45c41cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45c41abf90 con 0x7f45c40a1740 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f45c0003610 con 0x7f45c40a1740 2024-09-16T10:29:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.137+0000 7f45cac66640 1 -- 172.21.15.94:0/3379963626 >> v1:172.21.15.152:6790/0 conn(0x7f45c40a42c0 legacy=0x7f45c41d3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient(hunting): dump: 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046429599 0 0) 0x7f45c41aaa70 con 0x7f45c40a37d0 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient(hunting): my global_id is 4294 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a8003520 con 0x7f45c40a37d0 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3015107609 0 0) 0x7f45c41abf90 con 0x7f45c40a1740 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 >> v1:172.21.15.152:6790/0 conn(0x7f45c40a42c0 legacy=0x7f45c41d3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 >> v1:172.21.15.152:6789/0 conn(0x7f45c40a37d0 legacy=0x7f45c41cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 monclient: found mon.a 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c41abcb0 con 0x7f45c40a1740 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45cc6c9640 5 monclient: authenticate success, global_id 4338 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45cc6c9640 10 monclient: _renew_subs 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45cc6c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c41aa7a0 con 0x7f45c40a1740 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45cc6c9640 10 monclient: _renew_subs 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45cc6c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c41befc0 con 0x7f45c40a1740 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45c0003c50 con 0x7f45c40a1740 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f45c0004100 con 0x7f45c40a1740 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45c8c62640 4 set_mon_vals no callback set 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 10 monclient: dump: 2024-09-16T10:29:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.155 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.138+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f45c00042e0 con 0x7f45c40a1740 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.139+0000 7f45cc6c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.139+0000 7f45cc6c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.139+0000 7f45cc6c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.139+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac005040 con 0x7f45c40a1740 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.139+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f45c00461e0 con 0x7f45c40a1740 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.139+0000 7f45c9c64640 10 monclient: get_auth_request con 0x7f45a8039030 auth_method 0 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.143+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f45c0015660 con 0x7f45c40a1740 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.143+0000 7f45baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.143+0000 7f45baffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:36.201 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:29:36.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T10:29:36.226 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:29:36.238 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T10:29:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.320+0000 7f45cc6c9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:29:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.320+0000 7f45cc6c9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:29:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.320+0000 7f45cc6c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.320+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f45ac0052f0 con 0x7f45c40a1740 2024-09-16T10:29:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.320+0000 7f45baffd640 1 -- 172.21.15.94:0/3379963626 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 956180462) 0x7f45c0013ad0 con 0x7f45c40a1740 2024-09-16T10:29:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.321+0000 7f45baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:29:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.321+0000 7f45baffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:36.323 INFO:teuthology.orchestra.run.smithi094.stdout:51539607555 2024-09-16T10:29:36.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.322+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 >> v1:172.21.15.152:6800/2686360781 conn(0x7f45a8039030 legacy=0x7f45a803b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.322+0000 7f45cc6c9640 10 monclient: shutdown 2024-09-16T10:29:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.322+0000 7f45cc6c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.322+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 >> v1:172.21.15.94:6789/0 conn(0x7f45c40a1740 legacy=0x7f45c41be8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.323+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 shutdown_connections 2024-09-16T10:29:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.323+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 >> 172.21.15.94:0/3379963626 conn(0x7f45c40a9460 msgr2=0x7f45c40a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.323+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 shutdown_connections 2024-09-16T10:29:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.323+0000 7f45cc6c9640 1 -- 172.21.15.94:0/3379963626 wait complete. 2024-09-16T10:29:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.328+0000 7f8c7904f640 10 monclient: get_monmap_and_config 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.328+0000 7f8c7904f640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.328+0000 7f8c7904f640 10 monclient: monmap: 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:36.329085+0000 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:36.329085+0000 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.331 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.329+0000 7f8c7904f640 1 Processor -- start 2024-09-16T10:29:36.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.329+0000 7f8c7904f640 1 -- start start 2024-09-16T10:29:36.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.329+0000 7f8c7904f640 10 monclient: init 2024-09-16T10:29:36.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.329+0000 7f8c7904f640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.329+0000 7f8c7904f640 10 monclient: _add_conns ranks=[4,6,3,7,8,2,0,5,1] 2024-09-16T10:29:36.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.329+0000 7f8c7904f640 10 monclient(hunting): picked mon.noname-c con 0x7f8c740a38d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:36.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.330+0000 7f8c7904f640 10 monclient(hunting): picked mon.noname-d con 0x7f8c7406c670 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:36.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.330+0000 7f8c7904f640 10 monclient(hunting): picked mon.noname-g con 0x7f8c740a43c0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:36.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.330+0000 7f8c72d9d640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f8c740a38d0 0x7f8c740a3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43406/0 (socket says 172.21.15.94:43406) 2024-09-16T10:29:36.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.330+0000 7f8c72d9d640 1 -- 172.21.15.94:0/2723024980 learned_addr learned my addr 172.21.15.94:0/2723024980 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.330+0000 7f8c7904f640 1 -- 172.21.15.94:0/2723024980 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c74061ae0 con 0x7f8c740a43c0 2024-09-16T10:29:36.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.330+0000 7f8c7904f640 1 -- 172.21.15.94:0/2723024980 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c740a04d0 con 0x7f8c740a38d0 2024-09-16T10:29:36.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.330+0000 7f8c7904f640 1 -- 172.21.15.94:0/2723024980 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c740a3250 con 0x7f8c7406c670 2024-09-16T10:29:36.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.330+0000 7f8c7904f640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.330+0000 7f8c7904f640 10 monclient(hunting): authenticate will time out at 743.670373s 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c72d9d640 1 -- 172.21.15.94:0/2723024980 >> v1:172.21.15.94:6791/0 conn(0x7f8c740a38d0 legacy=0x7f8c740a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c7259c640 1 -- 172.21.15.94:0/2723024980 >> v1:172.21.15.94:6792/0 conn(0x7f8c7406c670 legacy=0x7f8c7413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c7359e640 1 -- 172.21.15.94:0/2723024980 >> v1:172.21.15.152:6790/0 conn(0x7f8c740a43c0 legacy=0x7f8c7413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c7359e640 1 -- 172.21.15.94:0/2723024980 >> v1:172.21.15.152:6790/0 conn(0x7f8c740a43c0 legacy=0x7f8c7413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c7359e640 1 -- 172.21.15.94:0/2723024980 >> v1:172.21.15.152:6790/0 conn(0x7f8c740a43c0 legacy=0x7f8c7413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c7359e640 1 --1- 172.21.15.94:0/2723024980 >> v1:172.21.15.152:6790/0 conn(0x7f8c740a43c0 0x7f8c7413f210 :-1 s=OPENED pgs=61 cs=1 l=1).handle_message read tag failed 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c7359e640 1 --1- 172.21.15.94:0/2723024980 >> v1:172.21.15.152:6790/0 conn(0x7f8c740a43c0 0x7f8c7413f210 :-1 s=OPENED pgs=61 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029674427 0 0) 0x7f8c740a04d0 con 0x7f8c740a38d0 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 10 monclient(hunting): my global_id is 4306 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c5c003520 con 0x7f8c740a38d0 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.152:6790/0 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164542180 0 0) 0x7f8c740a3250 con 0x7f8c7406c670 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 10 monclient(hunting): my global_id is 4290 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.332+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c740a04d0 con 0x7f8c7406c670 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3937812563 0 0) 0x7f8c740a04d0 con 0x7f8c7406c670 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 >> v1:172.21.15.94:6791/0 conn(0x7f8c740a38d0 legacy=0x7f8c740a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 >> v1:172.21.15.152:6790/0 conn(0x7f8c740a43c0 legacy=0x7f8c7413f210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 1 monclient: found mon.noname-d 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c740a0a00 con 0x7f8c7406c670 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8c680019c0 con 0x7f8c7406c670 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: dump: 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.338 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c7904f640 5 monclient: authenticate success, global_id 4290 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c7904f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c68001cd0 con 0x7f8c7406c670 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 1 -- 172.21.15.94:0/2723024980 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8c68003950 con 0x7f8c7406c670 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c7904f640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c7904f640 4 set_mon_vals no callback set 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.333+0000 7f8c71d9b640 10 monclient: dump: 2024-09-16T10:29:36.339 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.340 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.341 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.341 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.334+0000 7f8c7904f640 10 monclient: shutdown 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.334+0000 7f8c7904f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.334+0000 7f8c7904f640 1 -- 172.21.15.94:0/2723024980 >> v1:172.21.15.94:6792/0 conn(0x7f8c7406c670 legacy=0x7f8c7413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.334+0000 7f8c7904f640 1 -- 172.21.15.94:0/2723024980 shutdown_connections 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.334+0000 7f8c7904f640 1 -- 172.21.15.94:0/2723024980 >> 172.21.15.94:0/2723024980 conn(0x7f8c740a59d0 msgr2=0x7f8c740a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.334+0000 7f8c7904f640 1 -- 172.21.15.94:0/2723024980 shutdown_connections 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.334+0000 7f8c7904f640 1 -- 172.21.15.94:0/2723024980 wait complete. 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.335+0000 7f8c7904f640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.335+0000 7f8c7904f640 10 monclient: monmap: 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.343 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.344 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607555 for osd.2 2024-09-16T10:29:36.345 DEBUG:teuthology.parallel:result is None 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.335+0000 7f8c7904f640 1 Processor -- start 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.335+0000 7f8c7904f640 1 -- start start 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.335+0000 7f8c7904f640 10 monclient: init 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 10 monclient: _add_conns ranks=[0,6,7,5,1,4,2,8,3] 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 10 monclient(hunting): picked mon.noname-a con 0x7f8c7406c670 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 10 monclient(hunting): picked mon.noname-d con 0x7f8c740a38d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 10 monclient(hunting): picked mon.noname-i con 0x7f8c740a43c0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7409ef60 con 0x7f8c7406c670 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7259c640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f8c740a38d0 0x7f8c741cfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60296/0 (socket says 172.21.15.94:60296) 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c72d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8c7406c670 0x7f8c74136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41490/0 (socket says 172.21.15.94:41490) 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7259c640 1 -- 172.21.15.94:0/624402448 learned_addr learned my addr 172.21.15.94:0/624402448 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c741255c0 con 0x7f8c740a38d0 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7406ba00 con 0x7f8c740a43c0 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.336+0000 7f8c7904f640 10 monclient(hunting): authenticate will time out at 743.676628s 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c7259c640 1 -- 172.21.15.94:0/624402448 >> v1:172.21.15.94:6792/0 conn(0x7f8c740a38d0 legacy=0x7f8c741cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c72d9d640 1 -- 172.21.15.94:0/624402448 >> v1:172.21.15.94:6789/0 conn(0x7f8c7406c670 legacy=0x7f8c74136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597240568 0 0) 0x7f8c741255c0 con 0x7f8c740a38d0 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 10 monclient(hunting): my global_id is 4299 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60003520 con 0x7f8c740a38d0 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814509389 0 0) 0x7f8c7409ef60 con 0x7f8c7406c670 2024-09-16T10:29:36.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 10 monclient(hunting): my global_id is 4347 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.338+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c741255c0 con 0x7f8c7406c670 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c7359e640 1 -- 172.21.15.94:0/624402448 >> v1:172.21.15.152:6792/0 conn(0x7f8c740a43c0 legacy=0x7f8c741d35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894122159 0 0) 0x7f8c60003520 con 0x7f8c740a38d0 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c7409ef60 con 0x7f8c740a38d0 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8c68003950 con 0x7f8c740a38d0 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): dump: 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.350 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177919077 0 0) 0x7f8c7406ba00 con 0x7f8c740a43c0 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): my global_id is 4354 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60003520 con 0x7f8c740a43c0 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3119767952 0 0) 0x7f8c741255c0 con 0x7f8c7406c670 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c7406ba00 con 0x7f8c7406c670 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8c640047e0 con 0x7f8c7406c670 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient(hunting): dump: 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.352 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.353 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 702331997 0 0) 0x7f8c7409ef60 con 0x7f8c740a38d0 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 >> v1:172.21.15.152:6792/0 conn(0x7f8c740a43c0 legacy=0x7f8c741d35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 >> v1:172.21.15.94:6789/0 conn(0x7f8c7406c670 legacy=0x7f8c74136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 monclient: found mon.d 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74127450 con 0x7f8c740a38d0 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c7904f640 5 monclient: authenticate success, global_id 4299 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c7904f640 10 monclient: _renew_subs 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c7904f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c74126bb0 con 0x7f8c740a38d0 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c68003ed0 con 0x7f8c740a38d0 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8c68004380 con 0x7f8c740a38d0 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c5b7fe640 10 monclient: dump: 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.355 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.339+0000 7f8c7159a640 4 set_mon_vals no callback set 2024-09-16T10:29:36.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.340+0000 7f8c7904f640 10 monclient: _renew_subs 2024-09-16T10:29:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.340+0000 7f8c7904f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.340+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7409f290 con 0x7f8c740a38d0 2024-09-16T10:29:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.341+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f8c680106c0 con 0x7f8c740a38d0 2024-09-16T10:29:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.341+0000 7f8c7904f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.341+0000 7f8c7904f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.341+0000 7f8c7904f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.341+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c74069e70 con 0x7f8c740a38d0 2024-09-16T10:29:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.341+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f8c68046460 con 0x7f8c740a38d0 2024-09-16T10:29:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.341+0000 7f8c72d9d640 10 monclient: get_auth_request con 0x7f8c600399d0 auth_method 0 2024-09-16T10:29:36.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.346+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f8c680157c0 con 0x7f8c740a38d0 2024-09-16T10:29:36.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.346+0000 7f8c5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.346+0000 7f8c5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:36.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.352+0000 7fc4e0861640 10 monclient: get_monmap_and_config 2024-09-16T10:29:36.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.352+0000 7fc4e0861640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.352+0000 7fc4e0861640 10 monclient: monmap: 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:36.353413+0000 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:36.353413+0000 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.359 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7f6e6647d640 10 monclient: get_monmap_and_config 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7f6e6647d640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7f6e6647d640 10 monclient: monmap: 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.360 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:36.353891+0000 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:36.353891+0000 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.362 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7fc4e0861640 1 Processor -- start 2024-09-16T10:29:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7fc4e0861640 1 -- start start 2024-09-16T10:29:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7fc4e0861640 10 monclient: init 2024-09-16T10:29:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7f6e6647d640 1 Processor -- start 2024-09-16T10:29:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7fc4e0861640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7fc4e0861640 10 monclient: _add_conns ranks=[5,1,4,6,8,2,0,7,3] 2024-09-16T10:29:36.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7fc4e0861640 10 monclient(hunting): picked mon.noname-h con 0x7fc4dc0a38d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:36.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7fc4e0861640 10 monclient(hunting): picked mon.noname-f con 0x7fc4dc06c670 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:36.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7fc4e0861640 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc0a43c0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:36.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7fc4e0861640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc061ae0 con 0x7fc4dc06c670 2024-09-16T10:29:36.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7fc4e0861640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc0a04d0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7f6e6647d640 1 -- start start 2024-09-16T10:29:36.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.353+0000 7f6e6647d640 10 monclient: init 2024-09-16T10:29:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7fc4e0861640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc0a3250 con 0x7fc4dc0a38d0 2024-09-16T10:29:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7fc4e0861640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7fc4dad9d640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fc4dc0a43c0 0x7fc4dc13f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43414/0 (socket says 172.21.15.94:43414) 2024-09-16T10:29:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7fc4dad9d640 1 -- 172.21.15.94:0/977132310 learned_addr learned my addr 172.21.15.94:0/977132310 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 10 monclient: _add_conns ranks=[3,2,6,4,1,8,7,0,5] 2024-09-16T10:29:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 10 monclient(hunting): picked mon.noname-g con 0x7f6e600a14c0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e600a1fb0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:36.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7fc4e0861640 10 monclient(hunting): authenticate will time out at 743.694292s 2024-09-16T10:29:36.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 10 monclient(hunting): picked mon.noname-d con 0x7f6e6006f0e0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:36.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e60061ae0 con 0x7f6e600a1fb0 2024-09-16T10:29:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e5f7fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f6e600a1fb0 0x7f6e6013fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48404/0 (socket says 172.21.15.94:48404) 2024-09-16T10:29:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e5f7fe640 1 -- 172.21.15.94:0/12624093 learned_addr learned my addr 172.21.15.94:0/12624093 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 1 -- 172.21.15.94:0/12624093 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600a0000 con 0x7f6e600a14c0 2024-09-16T10:29:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 1 -- 172.21.15.94:0/12624093 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6006e0f0 con 0x7f6e6006f0e0 2024-09-16T10:29:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.354+0000 7f6e6647d640 10 monclient(hunting): authenticate will time out at 743.694609s 2024-09-16T10:29:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4dad9d640 1 -- 172.21.15.94:0/977132310 >> v1:172.21.15.94:6791/0 conn(0x7fc4dc0a43c0 legacy=0x7fc4dc13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d9d9b640 1 -- 172.21.15.94:0/977132310 >> v1:172.21.15.152:6789/0 conn(0x7fc4dc06c670 legacy=0x7fc4dc13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4da59c640 1 -- 172.21.15.94:0/977132310 >> v1:172.21.15.152:6791/0 conn(0x7fc4dc0a38d0 legacy=0x7fc4dc0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5f7fe640 1 -- 172.21.15.94:0/12624093 >> v1:172.21.15.94:6790/0 conn(0x7f6e600a1fb0 legacy=0x7f6e6013fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989876121 0 0) 0x7fc4dc0a04d0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): my global_id is 4315 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c4003520 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196348435 0 0) 0x7fc4dc061ae0 con 0x7fc4dc06c670 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): my global_id is 4303 2024-09-16T10:29:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4dc0a04d0 con 0x7fc4dc06c670 2024-09-16T10:29:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5ffff640 1 -- 172.21.15.94:0/12624093 >> v1:172.21.15.152:6790/0 conn(0x7f6e600a14c0 legacy=0x7f6e600a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042477662 0 0) 0x7fc4dc0a3250 con 0x7fc4dc0a38d0 2024-09-16T10:29:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e64a1a640 1 -- 172.21.15.94:0/12624093 >> v1:172.21.15.94:6792/0 conn(0x7f6e6006f0e0 legacy=0x7f6e601433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 10 monclient(hunting): my global_id is 4334 2024-09-16T10:29:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4dc061ae0 con 0x7fc4dc0a38d0 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121023858 0 0) 0x7f6e60061ae0 con 0x7f6e600a1fb0 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 10 monclient(hunting): my global_id is 4304 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48003520 con 0x7f6e600a1fb0 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436644376 0 0) 0x7f6e500013d0 con 0x7f6e600a14c0 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2248501902 0 0) 0x7fc4c4003520 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 10 monclient(hunting): my global_id is 4377 2024-09-16T10:29:36.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.356+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e60061ae0 con 0x7f6e600a14c0 2024-09-16T10:29:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 >> v1:172.21.15.152:6791/0 conn(0x7fc4dc0a38d0 legacy=0x7fc4dc0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 >> v1:172.21.15.152:6789/0 conn(0x7fc4dc06c670 legacy=0x7fc4dc13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 1 monclient: found mon.noname-c 2024-09-16T10:29:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:29:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc0a0a00 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4d4001ea0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: dump: 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4e0861640 5 monclient: authenticate success, global_id 4315 2024-09-16T10:29:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4e0861640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:36.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089733943 0 0) 0x7f6e6006e0f0 con 0x7f6e6006f0e0 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient(hunting): my global_id is 4308 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e500013d0 con 0x7f6e6006f0e0 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 448376106 0 0) 0x7f6e48003520 con 0x7f6e600a1fb0 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 >> v1:172.21.15.94:6792/0 conn(0x7f6e6006f0e0 legacy=0x7f6e601433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 >> v1:172.21.15.152:6790/0 conn(0x7f6e600a14c0 legacy=0x7f6e600a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 monclient: found mon.noname-b 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6006de10 con 0x7f6e600a1fb0 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6e54001aa0 con 0x7f6e600a1fb0 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: dump: 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.379 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e6647d640 5 monclient: authenticate success, global_id 4304 2024-09-16T10:29:36.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e6647d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:36.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4d4004c20 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4e0861640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:36.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4e0861640 4 set_mon_vals no callback set 2024-09-16T10:29:36.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 1 -- 172.21.15.94:0/977132310 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4d4005190 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e54001cc0 con 0x7f6e600a1fb0 2024-09-16T10:29:36.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e6647d640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:36.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e6647d640 4 set_mon_vals no callback set 2024-09-16T10:29:36.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 1 -- 172.21.15.94:0/12624093 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6e54004ca0 con 0x7f6e600a1fb0 2024-09-16T10:29:36.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7f6e5effd640 10 monclient: dump: 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.383 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.384 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.384 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.384 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.384 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.384 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.384 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.384 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.357+0000 7fc4d959a640 10 monclient: dump: 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.385 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7fc4e0861640 10 monclient: shutdown 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7fc4e0861640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7fc4e0861640 1 -- 172.21.15.94:0/977132310 >> v1:172.21.15.94:6791/0 conn(0x7fc4dc0a43c0 legacy=0x7fc4dc13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7f6e6647d640 10 monclient: shutdown 2024-09-16T10:29:36.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7f6e6647d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7f6e6647d640 1 -- 172.21.15.94:0/12624093 >> v1:172.21.15.94:6790/0 conn(0x7f6e600a1fb0 legacy=0x7f6e6013fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7f6e6647d640 1 -- 172.21.15.94:0/12624093 shutdown_connections 2024-09-16T10:29:36.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7f6e6647d640 1 -- 172.21.15.94:0/12624093 >> 172.21.15.94:0/12624093 conn(0x7f6e600a2ec0 msgr2=0x7f6e60065c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7fc4e0861640 1 -- 172.21.15.94:0/977132310 shutdown_connections 2024-09-16T10:29:36.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7fc4e0861640 1 -- 172.21.15.94:0/977132310 >> 172.21.15.94:0/977132310 conn(0x7fc4dc0a59d0 msgr2=0x7fc4dc0a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7f6e6647d640 1 -- 172.21.15.94:0/12624093 shutdown_connections 2024-09-16T10:29:36.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7fc4e0861640 1 -- 172.21.15.94:0/977132310 shutdown_connections 2024-09-16T10:29:36.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7f6e6647d640 1 -- 172.21.15.94:0/12624093 wait complete. 2024-09-16T10:29:36.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7fc4e0861640 1 -- 172.21.15.94:0/977132310 wait complete. 2024-09-16T10:29:36.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7f6e6647d640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.358+0000 7f6e6647d640 10 monclient: monmap: 2024-09-16T10:29:36.389 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.389 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:36.389 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:36.389 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:36.389 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.389 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.390 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7fc4e0861640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7fc4e0861640 10 monclient: monmap: 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.391 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.392 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.392 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.392 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.392 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.392 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7f6e6647d640 1 Processor -- start 2024-09-16T10:29:36.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7fc4e0861640 1 Processor -- start 2024-09-16T10:29:36.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7fc4e0861640 1 -- start start 2024-09-16T10:29:36.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7fc4e0861640 10 monclient: init 2024-09-16T10:29:36.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7f6e6647d640 1 -- start start 2024-09-16T10:29:36.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7f6e6647d640 10 monclient: init 2024-09-16T10:29:36.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7fc4e0861640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7fc4e0861640 10 monclient: _add_conns ranks=[0,4,1,5,6,8,3,2,7] 2024-09-16T10:29:36.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7f6e6647d640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7f6e6647d640 10 monclient: _add_conns ranks=[4,5,1,0,3,8,6,2,7] 2024-09-16T10:29:36.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7f6e6647d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e6006f0e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:36.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.359+0000 7fc4e0861640 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc06c670 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:36.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7f6e6647d640 10 monclient(hunting): picked mon.noname-h con 0x7f6e600a14c0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:36.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4e0861640 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc0a38d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:36.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7f6e5ffff640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f6e6006f0e0 0x7f6e601ced70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43428/0 (socket says 172.21.15.94:43428) 2024-09-16T10:29:36.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7f6e5ffff640 1 -- 172.21.15.94:0/2893679282 learned_addr learned my addr 172.21.15.94:0/2893679282 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4e0861640 10 monclient(hunting): picked mon.noname-f con 0x7fc4dc0a43c0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:36.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7f6e6647d640 10 monclient(hunting): picked mon.noname-f con 0x7f6e600a1fb0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:36.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4e0861640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc09ee60 con 0x7fc4dc06c670 2024-09-16T10:29:36.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4e0861640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc1370a0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4e0861640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc1265d0 con 0x7fc4dc0a38d0 2024-09-16T10:29:36.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4e0861640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4e0861640 10 monclient(hunting): authenticate will time out at 743.700183s 2024-09-16T10:29:36.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4d9d9b640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fc4dc0a38d0 0x7fc4dc1cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43420/0 (socket says 172.21.15.94:43420) 2024-09-16T10:29:36.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4da59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc4dc06c670 0x7fc4dc136990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41506/0 (socket says 172.21.15.94:41506) 2024-09-16T10:29:36.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7fc4d9d9b640 1 -- 172.21.15.94:0/315036876 learned_addr learned my addr 172.21.15.94:0/315036876 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6012ba00 con 0x7f6e600a1fb0 2024-09-16T10:29:36.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600709b0 con 0x7f6e6006f0e0 2024-09-16T10:29:36.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600a1ae0 con 0x7f6e600a14c0 2024-09-16T10:29:36.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7f6e6647d640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.360+0000 7f6e6647d640 10 monclient(hunting): authenticate will time out at 743.700399s 2024-09-16T10:29:36.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e64a1a640 1 -- 172.21.15.94:0/2893679282 >> v1:172.21.15.152:6789/0 conn(0x7f6e600a1fb0 legacy=0x7f6e601d70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4da59c640 1 -- 172.21.15.94:0/315036876 >> v1:172.21.15.94:6789/0 conn(0x7fc4dc06c670 legacy=0x7fc4dc136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4dad9d640 1 -- 172.21.15.94:0/315036876 >> v1:172.21.15.152:6789/0 conn(0x7fc4dc0a43c0 legacy=0x7fc4dc1d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5ffff640 1 -- 172.21.15.94:0/2893679282 >> v1:172.21.15.94:6791/0 conn(0x7f6e6006f0e0 legacy=0x7f6e601ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972386998 0 0) 0x7f6e6012ba00 con 0x7f6e600a1fb0 2024-09-16T10:29:36.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412395295 0 0) 0x7fc4dc09ee60 con 0x7fc4dc06c670 2024-09-16T10:29:36.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 10 monclient(hunting): my global_id is 4356 2024-09-16T10:29:36.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4bc003520 con 0x7fc4dc06c670 2024-09-16T10:29:36.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349406330 0 0) 0x7fc4dc1370a0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 10 monclient(hunting): my global_id is 4321 2024-09-16T10:29:36.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4dc09ee60 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5f7fe640 1 -- 172.21.15.94:0/2893679282 >> v1:172.21.15.152:6791/0 conn(0x7f6e600a14c0 legacy=0x7f6e6013b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 10 monclient(hunting): my global_id is 4312 2024-09-16T10:29:36.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c003520 con 0x7f6e600a1fb0 2024-09-16T10:29:36.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653589759 0 0) 0x7f6e600a1ae0 con 0x7f6e600a14c0 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2356623841 0 0) 0x7fc4dc09ee60 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4dc1370a0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4d40050d0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient(hunting): dump: 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.403 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:36.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2432410174 0 0) 0x7fc4bc003520 con 0x7fc4dc06c670 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4dc09ee60 con 0x7fc4dc06c670 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4cc001d50 con 0x7fc4dc06c670 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient(hunting): dump: 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.405 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.406 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.406 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.406 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.406 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.406 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.406 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.406 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3766798771 0 0) 0x7fc4dc1370a0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 >> v1:172.21.15.94:6791/0 conn(0x7fc4dc0a38d0 legacy=0x7fc4dc1cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 >> v1:172.21.15.94:6789/0 conn(0x7fc4dc06c670 legacy=0x7fc4dc136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 10 monclient(hunting): my global_id is 4343 2024-09-16T10:29:36.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.362+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e6012ba00 con 0x7f6e600a14c0 2024-09-16T10:29:36.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838900743 0 0) 0x7f6e6012ba00 con 0x7f6e600a14c0 2024-09-16T10:29:36.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e600a1ae0 con 0x7f6e600a14c0 2024-09-16T10:29:36.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6e54004b20 con 0x7f6e600a14c0 2024-09-16T10:29:36.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient(hunting): dump: 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.409 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3985247430 0 0) 0x7f6e4c003520 con 0x7f6e600a1fb0 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e6012ba00 con 0x7f6e600a1fb0 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6e580020d0 con 0x7f6e600a1fb0 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient(hunting): dump: 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.410 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 5320423 0 0) 0x7f6e600a1ae0 con 0x7f6e600a14c0 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 >> v1:172.21.15.94:6791/0 conn(0x7f6e6006f0e0 legacy=0x7f6e601ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 >> v1:172.21.15.152:6789/0 conn(0x7f6e600a1fb0 legacy=0x7f6e601d70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 monclient: found mon.h 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e60070f00 con 0x7f6e600a14c0 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e6647d640 5 monclient: authenticate success, global_id 4343 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e6647d640 10 monclient: _renew_subs 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e6647d640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:36.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6012b160 con 0x7f6e600a14c0 2024-09-16T10:29:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 monclient: found mon.f 2024-09-16T10:29:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc124d40 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4e0861640 5 monclient: authenticate success, global_id 4321 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4e0861640 10 monclient: _renew_subs 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4e0861640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc09f190 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4e0861640 10 monclient: _renew_subs 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4e0861640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc126e30 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4d40020d0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4d4001e80 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4caffd640 10 monclient: dump: 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.415 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.416 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7fc4d8d99640 4 set_mon_vals no callback set 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e6647d640 10 monclient: _renew_subs 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e6647d640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6013b720 con 0x7f6e600a14c0 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e54001a40 con 0x7f6e600a14c0 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6e540054c0 con 0x7f6e600a14c0 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5cff9640 10 monclient: dump: 2024-09-16T10:29:36.417 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.418 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.419 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.419 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.363+0000 7f6e5e7fc640 4 set_mon_vals no callback set 2024-09-16T10:29:36.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.364+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fc4d40108b0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.364+0000 7f6e6647d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.365+0000 7f6e6647d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.364+0000 7fc4e0861640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.364+0000 7fc4e0861640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.364+0000 7fc4e0861640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:36.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.365+0000 7f6e6647d640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:36.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.364+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c0005040 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.365+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7fc4d4046270 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.365+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e600707b0 con 0x7f6e600a14c0 2024-09-16T10:29:36.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.365+0000 7fc4da59c640 10 monclient: get_auth_request con 0x7fc4bc039680 auth_method 0 2024-09-16T10:29:36.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.365+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f6e54010520 con 0x7f6e600a14c0 2024-09-16T10:29:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.365+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f6e540460a0 con 0x7f6e600a14c0 2024-09-16T10:29:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.367+0000 7f6e5ffff640 10 monclient: get_auth_request con 0x7f6e4c0397f0 auth_method 0 2024-09-16T10:29:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.370+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f6e54015410 con 0x7f6e600a14c0 2024-09-16T10:29:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.370+0000 7f6e5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.370+0000 7f6e5cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.370+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fc4d40155e0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.370+0000 7fc4caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.370+0000 7fc4caffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:36.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.371+0000 7f1fae708640 10 monclient: get_monmap_and_config 2024-09-16T10:29:36.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.371+0000 7f1fae708640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.371+0000 7f1fae708640 10 monclient: monmap: 2024-09-16T10:29:36.432 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.432 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:36.372122+0000 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:36.372122+0000 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.433 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.372+0000 7f1fae708640 1 Processor -- start 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.372+0000 7f1fae708640 1 -- start start 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.372+0000 7f1fae708640 10 monclient: init 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.372+0000 7f1fae708640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.372+0000 7f1fae708640 10 monclient: _add_conns ranks=[2,8,0,3,1,5,6,4,7] 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.372+0000 7f1fae708640 10 monclient(hunting): picked mon.noname-b con 0x7f1fa80a38d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fae708640 10 monclient(hunting): picked mon.noname-e con 0x7f1fa806c670 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fa77fe640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f1fa806c670 0x7f1fa813bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33542/0 (socket says 172.21.15.94:33542) 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fa7fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f1fa80a38d0 0x7f1fa80a3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48416/0 (socket says 172.21.15.94:48416) 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1facca5640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1fa80a43c0 0x7f1fa813f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41510/0 (socket says 172.21.15.94:41510) 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fae708640 10 monclient(hunting): picked mon.noname-a con 0x7f1fa80a43c0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fa77fe640 1 -- 172.21.15.94:0/3034858464 learned_addr learned my addr 172.21.15.94:0/3034858464 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fae708640 1 -- 172.21.15.94:0/3034858464 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa8061ae0 con 0x7f1fa80a43c0 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fae708640 1 -- 172.21.15.94:0/3034858464 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa80a04d0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fae708640 1 -- 172.21.15.94:0/3034858464 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa80a3250 con 0x7f1fa806c670 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fae708640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.373+0000 7f1fae708640 10 monclient(hunting): authenticate will time out at 743.713420s 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa77fe640 1 -- 172.21.15.94:0/3034858464 >> v1:172.21.15.94:6793/0 conn(0x7f1fa806c670 legacy=0x7f1fa813bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1facca5640 1 -- 172.21.15.94:0/3034858464 >> v1:172.21.15.94:6789/0 conn(0x7f1fa80a43c0 legacy=0x7f1fa813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa7fff640 1 -- 172.21.15.94:0/3034858464 >> v1:172.21.15.94:6790/0 conn(0x7f1fa80a38d0 legacy=0x7f1fa80a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399219904 0 0) 0x7f1fa80a3250 con 0x7f1fa806c670 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): my global_id is 4310 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f90003520 con 0x7f1fa806c670 2024-09-16T10:29:36.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392926805 0 0) 0x7f1fa8061ae0 con 0x7f1fa80a43c0 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): my global_id is 4365 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa80a3250 con 0x7f1fa80a43c0 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359834670 0 0) 0x7f1fa80a04d0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 10 monclient(hunting): my global_id is 4313 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.375+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa8061ae0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa77fe640 1 --1- 172.21.15.94:0/3034858464 >> v1:172.21.15.94:6793/0 conn(0x7f1fa806c670 0x7f1fa813bae0 :-1 s=OPENED pgs=59 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7f1f9c001a60 mon_map magic: 0 v1 2024-09-16T10:29:36.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 417407952 0 0) 0x7f1fa80a3250 con 0x7f1fa80a43c0 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 >> v1:172.21.15.94:6793/0 conn(0x7f1fa806c670 legacy=0x7f1fa813bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 >> v1:172.21.15.94:6790/0 conn(0x7f1fa80a38d0 legacy=0x7f1fa80a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 1 monclient: found mon.noname-a 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa80a0a00 con 0x7f1fa80a43c0 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1fa0001d60 con 0x7f1fa80a43c0 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:36.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: dump: 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.438 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fae708640 5 monclient: authenticate success, global_id 4365 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1f9c001a60 con 0x7f1fa806c670 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fae708640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fa0002070 con 0x7f1fa80a43c0 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 1 -- 172.21.15.94:0/3034858464 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1fa0004ea0 con 0x7f1fa80a43c0 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fae708640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fae708640 4 set_mon_vals no callback set 2024-09-16T10:29:36.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.376+0000 7f1fa6ffd640 10 monclient: dump: 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.377+0000 7f1fae708640 10 monclient: shutdown 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.377+0000 7f1fae708640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.377+0000 7f1fae708640 1 -- 172.21.15.94:0/3034858464 >> v1:172.21.15.94:6789/0 conn(0x7f1fa80a43c0 legacy=0x7f1fa813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.377+0000 7f1fae708640 1 -- 172.21.15.94:0/3034858464 shutdown_connections 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.377+0000 7f1fae708640 1 -- 172.21.15.94:0/3034858464 >> 172.21.15.94:0/3034858464 conn(0x7f1fa80a59d0 msgr2=0x7f1fa80a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.377+0000 7f1fae708640 1 -- 172.21.15.94:0/3034858464 shutdown_connections 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.377+0000 7f1fae708640 1 -- 172.21.15.94:0/3034858464 wait complete. 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.378+0000 7f1fae708640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.378+0000 7f1fae708640 10 monclient: monmap: 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.442 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.379+0000 7f1fae708640 1 Processor -- start 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.379+0000 7f1fae708640 1 -- start start 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.379+0000 7f1fae708640 10 monclient: init 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.380+0000 7f1fae708640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.380+0000 7f1fae708640 10 monclient: _add_conns ranks=[5,4,1,2,7,8,6,0,3] 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.380+0000 7f1fae708640 10 monclient(hunting): picked mon.noname-h con 0x7f1fa806c670 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.380+0000 7f1fae708640 10 monclient(hunting): picked mon.noname-c con 0x7f1fa80a38d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.380+0000 7f1fae708640 10 monclient(hunting): picked mon.noname-f con 0x7f1fa80a43c0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.380+0000 7f1fae708640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa809f320 con 0x7f1fa80a43c0 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.380+0000 7f1fae708640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa8125d60 con 0x7f1fa80a38d0 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.380+0000 7f1fa77fe640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f1fa80a38d0 0x7f1fa81cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43430/0 (socket says 172.21.15.94:43430) 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.380+0000 7f1fa77fe640 1 -- 172.21.15.94:0/4189720047 learned_addr learned my addr 172.21.15.94:0/4189720047 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.381+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa809ef80 con 0x7f1fa806c670 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.381+0000 7f1fae708640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.381+0000 7f1fae708640 10 monclient(hunting): authenticate will time out at 743.721109s 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa77fe640 1 -- 172.21.15.94:0/4189720047 >> v1:172.21.15.94:6791/0 conn(0x7f1fa80a38d0 legacy=0x7f1fa81cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa7fff640 1 -- 172.21.15.94:0/4189720047 >> v1:172.21.15.152:6791/0 conn(0x7f1fa806c670 legacy=0x7f1fa81368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020340764 0 0) 0x7f1fa8125d60 con 0x7f1fa80a38d0 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): my global_id is 4324 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1facca5640 1 -- 172.21.15.94:0/4189720047 >> v1:172.21.15.152:6789/0 conn(0x7f1fa80a43c0 legacy=0x7f1fa81d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f94003520 con 0x7f1fa80a38d0 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010253947 0 0) 0x7f1fa80a0a00 con 0x7f1fa80a43c0 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): my global_id is 4330 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa8125d60 con 0x7f1fa80a43c0 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630748922 0 0) 0x7f1fa809ef80 con 0x7f1fa806c670 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): my global_id is 4352 2024-09-16T10:29:36.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa80a0a00 con 0x7f1fa806c670 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3081311743 0 0) 0x7f1f94003520 con 0x7f1fa80a38d0 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fa809ef80 con 0x7f1fa80a38d0 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1f9c001320 con 0x7f1fa80a38d0 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): dump: 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.446 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:36.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2440920305 0 0) 0x7f1fa809ef80 con 0x7f1fa80a38d0 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 >> v1:172.21.15.152:6791/0 conn(0x7f1fa806c670 legacy=0x7f1fa81368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 >> v1:172.21.15.152:6789/0 conn(0x7f1fa80a43c0 legacy=0x7f1fa81d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 monclient: found mon.c 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa806ba10 con 0x7f1fa80a38d0 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fa4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fae708640 5 monclient: authenticate success, global_id 4324 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fae708640 10 monclient: _renew_subs 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fae708640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:36.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.383+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa81272d0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fae708640 10 monclient: _renew_subs 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fae708640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa8124f70 con 0x7f1fa80a38d0 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f9c004dd0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fa4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1f9c005280 con 0x7f1fa80a38d0 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fa4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fa4ff9640 10 monclient: dump: 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fa67fc640 4 set_mon_vals no callback set 2024-09-16T10:29:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.384+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f1f9c0102e0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.385+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f1f9c045fe0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.385+0000 7f1fa7fff640 10 monclient: get_auth_request con 0x7f1f94039310 auth_method 0 2024-09-16T10:29:36.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.385+0000 7f1fae708640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.385+0000 7f1fae708640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.385+0000 7f1fae708640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:36.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.385+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa81270d0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.390+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f1f9c0154c0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.390+0000 7f1fa4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.390+0000 7f1fa4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f6e6647d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:29:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f6e6647d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:29:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f6e6647d640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:29:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 --> v1:172.21.15.152:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6e6006b030 con 0x7f6e600a14c0 2024-09-16T10:29:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7fc4e0861640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:29:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7fc4e0861640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:29:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7fc4e0861640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:29:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc4c00052f0 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f6e5cff9640 1 -- 172.21.15.94:0/2893679282 <== mon.5 v1:172.21.15.152:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 1444633209) 0x7f6e54015230 con 0x7f6e600a14c0 2024-09-16T10:29:36.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f6e5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:29:36.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f6e5cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:36.544 INFO:teuthology.orchestra.run.smithi094.stdout:47244640260 2024-09-16T10:29:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f8c7904f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:29:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f8c7904f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:29:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f8c7904f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8c7406ac30 con 0x7f8c740a38d0 2024-09-16T10:29:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.541+0000 7fc4caffd640 1 -- 172.21.15.94:0/315036876 <== mon.1 v1:172.21.15.152:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 847495621) 0x7fc4d4046020 con 0x7fc4dc0a43c0 2024-09-16T10:29:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.542+0000 7fc4caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:29:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.542+0000 7fc4caffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:36.546 INFO:teuthology.orchestra.run.smithi094.stdout:47244640259 2024-09-16T10:29:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.542+0000 7f8c5b7fe640 1 -- 172.21.15.94:0/624402448 <== mon.6 v1:172.21.15.94:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 847495621) 0x7f8c68013c30 con 0x7f8c740a38d0 2024-09-16T10:29:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.542+0000 7f8c5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:29:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.542+0000 7f8c5b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:36.547 INFO:teuthology.orchestra.run.smithi094.stdout:47244640259 2024-09-16T10:29:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 >> v1:172.21.15.152:6800/2686360781 conn(0x7f6e4c0397f0 legacy=0x7f6e4c03bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7f6e6647d640 10 monclient: shutdown 2024-09-16T10:29:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7f6e6647d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 >> v1:172.21.15.152:6791/0 conn(0x7f6e600a14c0 legacy=0x7f6e6013b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 shutdown_connections 2024-09-16T10:29:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 >> 172.21.15.94:0/2893679282 conn(0x7f6e600a2ec0 msgr2=0x7f6e600a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 >> v1:172.21.15.152:6800/2686360781 conn(0x7fc4bc039680 legacy=0x7fc4bc03bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7fc4e0861640 10 monclient: shutdown 2024-09-16T10:29:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7fc4e0861640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 >> v1:172.21.15.152:6789/0 conn(0x7fc4dc0a43c0 legacy=0x7fc4dc1d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 shutdown_connections 2024-09-16T10:29:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 shutdown_connections 2024-09-16T10:29:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 >> 172.21.15.94:0/315036876 conn(0x7fc4dc0a59d0 msgr2=0x7fc4dc09ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7f6e6647d640 1 -- 172.21.15.94:0/2893679282 wait complete. 2024-09-16T10:29:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 shutdown_connections 2024-09-16T10:29:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.544+0000 7fc4e0861640 1 -- 172.21.15.94:0/315036876 wait complete. 2024-09-16T10:29:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.545+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8c600399d0 legacy=0x7f8c6003be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.545+0000 7f8c7904f640 10 monclient: shutdown 2024-09-16T10:29:36.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.545+0000 7f8c7904f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.545+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 >> v1:172.21.15.94:6792/0 conn(0x7f8c740a38d0 legacy=0x7f8c741cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.545+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 shutdown_connections 2024-09-16T10:29:36.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.545+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 >> 172.21.15.94:0/624402448 conn(0x7f8c740a59d0 msgr2=0x7f8c7409ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.545+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 shutdown_connections 2024-09-16T10:29:36.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.545+0000 7f8c7904f640 1 -- 172.21.15.94:0/624402448 wait complete. 2024-09-16T10:29:36.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-09-16T10:29:36.558 DEBUG:teuthology.parallel:result is None 2024-09-16T10:29:36.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-16T10:29:36.559 DEBUG:teuthology.parallel:result is None 2024-09-16T10:29:36.563 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2024-09-16T10:29:36.563 DEBUG:teuthology.parallel:result is None 2024-09-16T10:29:36.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.582+0000 7f1fae708640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:29:36.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.582+0000 7f1fae708640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:29:36.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.582+0000 7f1fae708640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:36.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.582+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f1fa80665a0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.582+0000 7f1fa4ff9640 1 -- 172.21.15.94:0/4189720047 <== mon.4 v1:172.21.15.94:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 847495621) 0x7f1f9c01dab0 con 0x7f1fa80a38d0 2024-09-16T10:29:36.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.582+0000 7f1fa4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:29:36.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.582+0000 7f1fa4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:36.585 INFO:teuthology.orchestra.run.smithi094.stdout:47244640259 2024-09-16T10:29:36.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.584+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 >> v1:172.21.15.152:6800/2686360781 conn(0x7f1f94039310 legacy=0x7f1f9403b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.584+0000 7f1fae708640 10 monclient: shutdown 2024-09-16T10:29:36.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.584+0000 7f1fae708640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.584+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 >> v1:172.21.15.94:6791/0 conn(0x7f1fa80a38d0 legacy=0x7f1fa81cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.584+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 shutdown_connections 2024-09-16T10:29:36.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.584+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 >> 172.21.15.94:0/4189720047 conn(0x7f1fa80a59d0 msgr2=0x7f1fa80a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.584+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 shutdown_connections 2024-09-16T10:29:36.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.584+0000 7f1fae708640 1 -- 172.21.15.94:0/4189720047 wait complete. 2024-09-16T10:29:36.596 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2024-09-16T10:29:36.596 DEBUG:teuthology.parallel:result is None 2024-09-16T10:29:36.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T10:29:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.949+0000 7f61c69bc640 10 monclient: get_monmap_and_config 2024-09-16T10:29:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.949+0000 7f61c69bc640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.949+0000 7f61c69bc640 10 monclient: monmap: 2024-09-16T10:29:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:36.950394+0000 2024-09-16T10:29:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:36.950394+0000 2024-09-16T10:29:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.952 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 1 Processor -- start 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 1 -- start start 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 10 monclient: init 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 10 monclient: _add_conns ranks=[2,3,4,6,5,0,7,1,8] 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 10 monclient(hunting): picked mon.noname-b con 0x7f61c00a37d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 10 monclient(hunting): picked mon.noname-g con 0x7f61c00a42c0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 10 monclient(hunting): picked mon.noname-c con 0x7f61c00a1740 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c0061ae0 con 0x7f61c00a37d0 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c00a4890 con 0x7f61c00a42c0 2024-09-16T10:29:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c006c490 con 0x7f61c00a1740 2024-09-16T10:29:36.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.950+0000 7f61c69bc640 10 monclient(hunting): authenticate will time out at 744.290992s 2024-09-16T10:29:36.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.951+0000 7f61bffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f61c00a37d0 0x7f61c00a3bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48426/0 (socket says 172.21.15.94:48426) 2024-09-16T10:29:36.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.951+0000 7f61c4f59640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f61c00a1740 0x7f61c013ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43438/0 (socket says 172.21.15.94:43438) 2024-09-16T10:29:36.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.951+0000 7f61bffff640 1 -- 172.21.15.94:0/1575133627 learned_addr learned my addr 172.21.15.94:0/1575133627 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bffff640 1 -- 172.21.15.94:0/1575133627 >> v1:172.21.15.94:6790/0 conn(0x7f61c00a37d0 legacy=0x7f61c00a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61c4f59640 1 -- 172.21.15.94:0/1575133627 >> v1:172.21.15.94:6791/0 conn(0x7f61c00a1740 legacy=0x7f61c013ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937006195 0 0) 0x7f61c0061ae0 con 0x7f61c00a37d0 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient(hunting): my global_id is 4322 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac003520 con 0x7f61c00a37d0 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411957406 0 0) 0x7f61c006c490 con 0x7f61c00a1740 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient(hunting): my global_id is 4333 2024-09-16T10:29:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c0061ae0 con 0x7f61c00a1740 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61b7fff640 1 -- 172.21.15.94:0/1575133627 >> v1:172.21.15.152:6790/0 conn(0x7f61c00a42c0 legacy=0x7f61c013b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677241287 0 0) 0x7f61ac003520 con 0x7f61c00a37d0 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 >> v1:172.21.15.94:6791/0 conn(0x7f61c00a1740 legacy=0x7f61c013ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 >> v1:172.21.15.152:6790/0 conn(0x7f61c00a42c0 legacy=0x7f61c013b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 monclient: found mon.noname-b 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c00a0d10 con 0x7f61c00a37d0 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f61b0001b20 con 0x7f61c00a37d0 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61c69bc640 5 monclient: authenticate success, global_id 4322 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61c69bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: dump: 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61c69bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61b00048a0 con 0x7f61c00a37d0 2024-09-16T10:29:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61c69bc640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 1 -- 172.21.15.94:0/1575133627 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f61b0004d50 con 0x7f61c00a37d0 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61c69bc640 4 set_mon_vals no callback set 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.953+0000 7f61bf7fe640 10 monclient: dump: 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.954+0000 7f61c69bc640 10 monclient: shutdown 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.954+0000 7f61c69bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.954+0000 7f61c69bc640 1 -- 172.21.15.94:0/1575133627 >> v1:172.21.15.94:6790/0 conn(0x7f61c00a37d0 legacy=0x7f61c00a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.954+0000 7f61c69bc640 1 -- 172.21.15.94:0/1575133627 shutdown_connections 2024-09-16T10:29:36.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.954+0000 7f61c69bc640 1 -- 172.21.15.94:0/1575133627 >> 172.21.15.94:0/1575133627 conn(0x7f61c00a9460 msgr2=0x7f61c00a58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.954+0000 7f61c69bc640 1 -- 172.21.15.94:0/1575133627 shutdown_connections 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.954+0000 7f61c69bc640 1 -- 172.21.15.94:0/1575133627 wait complete. 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.954+0000 7f61c69bc640 10 monclient: build_initial_monmap 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.954+0000 7f61c69bc640 10 monclient: monmap: 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 1 Processor -- start 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 1 -- start start 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 10 monclient: init 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 10 monclient: _add_conns ranks=[5,0,7,3,6,2,4,8,1] 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 10 monclient(hunting): picked mon.noname-h con 0x7f61c00a1740 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 10 monclient(hunting): picked mon.noname-a con 0x7f61c00a37d0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 10 monclient(hunting): picked mon.noname-i con 0x7f61c00a42c0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c00a7180 con 0x7f61c00a37d0 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c00a76d0 con 0x7f61c00a1740 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c009f560 con 0x7f61c00a42c0 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61c69bc640 10 monclient(hunting): authenticate will time out at 744.295915s 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61b7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f61c00a37d0 0x7f61c01cfb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41520/0 (socket says 172.21.15.94:41520) 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.955+0000 7f61b7fff640 1 -- 172.21.15.94:0/849300710 learned_addr learned my addr 172.21.15.94:0/849300710 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.956+0000 7f61c4f59640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6792/0 conn(0x7f61c00a42c0 legacy=0x7f61c01d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.956+0000 7f61c4f59640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6792/0 conn(0x7f61c00a42c0 legacy=0x7f61c01d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.956+0000 7f61c4f59640 1 --1- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6792/0 conn(0x7f61c00a42c0 0x7f61c01d32a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.956+0000 7f61c4f59640 1 --1- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6792/0 conn(0x7f61c00a42c0 0x7f61c01d32a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T10:29:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.956+0000 7f61bd7fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.152:6792/0 2024-09-16T10:29:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61b7fff640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.94:6789/0 conn(0x7f61c00a37d0 legacy=0x7f61c01cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241084542 0 0) 0x7f61c00a7180 con 0x7f61c00a37d0 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient(hunting): my global_id is 4374 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0003550 con 0x7f61c00a37d0 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bffff640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6791/0 conn(0x7f61c00a1740 legacy=0x7f61c01be670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 941988400 0 0) 0x7f61a0003550 con 0x7f61c00a37d0 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61c00a7180 con 0x7f61c00a37d0 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f61a8001d10 con 0x7f61c00a37d0 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient(hunting): dump: 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 969721931 0 0) 0x7f61c00a7180 con 0x7f61c00a37d0 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6792/0 conn(0x7f61c00a42c0 legacy=0x7f61c01d32a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6791/0 conn(0x7f61c00a1740 legacy=0x7f61c01be670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 monclient: found mon.a 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c01abd50 con 0x7f61c00a37d0 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:36.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61c69bc640 5 monclient: authenticate success, global_id 4374 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61c69bc640 10 monclient: _renew_subs 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61c69bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c01ac030 con 0x7f61c00a37d0 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61c69bc640 10 monclient: _renew_subs 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61c69bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c01bed80 con 0x7f61c00a37d0 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61a8003b30 con 0x7f61c00a37d0 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f61a8003fe0 con 0x7f61c00a37d0 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61bd7fa640 10 monclient: dump: 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.958+0000 7f61beffd640 4 set_mon_vals no callback set 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.959+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f61a80041c0 con 0x7f61c00a37d0 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.959+0000 7f61c69bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.959+0000 7f61c69bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.959+0000 7f61c69bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.959+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c01ab790 con 0x7f61c00a37d0 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.959+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f61a8045fd0 con 0x7f61c00a37d0 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.959+0000 7f61bffff640 10 monclient: get_auth_request con 0x7f61a0039090 auth_method 0 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.964+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f61a80154b0 con 0x7f61c00a37d0 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.964+0000 7f61bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:36.964+0000 7f61bd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:37.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.131+0000 7f61c69bc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:37.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61c69bc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:37.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61c69bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:29:37.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f61c01bef70 con 0x7f61c00a37d0 2024-09-16T10:29:37.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61b7fff640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.94:6789/0 conn(0x7f61c00a37d0 legacy=0x7f61c01cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:29:37.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61b7fff640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.94:6789/0 conn(0x7f61c00a37d0 legacy=0x7f61c01cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:29:37.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61b7fff640 1 --1- 172.21.15.94:0/849300710 >> v1:172.21.15.94:6789/0 conn(0x7f61c00a37d0 0x7f61c01cfb70 :-1 s=OPENED pgs=107 cs=1 l=1).handle_message read tag failed 2024-09-16T10:29:37.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61b7fff640 1 --1- 172.21.15.94:0/849300710 >> v1:172.21.15.94:6789/0 conn(0x7f61c00a37d0 0x7f61c01cfb70 :-1 s=OPENED pgs=107 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.94:6789/0 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.94:6789/0 conn(0x7f61c00a37d0 legacy=0x7f61c01cfb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 10 monclient: _add_conns ranks=[3,1,4,6,2,5,0,7,8] 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 10 monclient(hunting): picked mon.g con 0x7f61a003fa40 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 10 monclient(hunting): picked mon.f con 0x7f61a0043870 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 10 monclient(hunting): picked mon.c con 0x7f61a0047a20 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f61c00a7180 con 0x7f61a0043870 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f61a0003550 con 0x7f61a003fa40 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f61a004dbf0 con 0x7f61a0047a20 2024-09-16T10:29:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.132+0000 7f61bd7fa640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:37.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bffff640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.94:6791/0 conn(0x7f61a0047a20 legacy=0x7f61a0049ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008879145 0 0) 0x7f61a004dbf0 con 0x7f61a0047a20 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6791/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f61a004dd80 con 0x7f61a0047a20 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61b7fff640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6790/0 conn(0x7f61a003fa40 legacy=0x7f61a0041e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61c4f59640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6789/0 conn(0x7f61a0043870 legacy=0x7f61a0045d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1260246456 0 0) 0x7f61a004dd80 con 0x7f61a0047a20 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6790/0 conn(0x7f61a003fa40 legacy=0x7f61a0041e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6789/0 conn(0x7f61a0043870 legacy=0x7f61a0045d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 1 monclient: found mon.c 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61c4f59640 1 -- 172.21.15.94:0/849300710 reap_dead start 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f61a004ee70 con 0x7f61a0047a20 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f61a004dbf0 con 0x7f61a0047a20 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.134+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f61b00017a0 con 0x7f61a0047a20 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61bd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61bd7fa640 10 monclient: dump: 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.140 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61b0001a80 con 0x7f61a0047a20 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61beffd640 4 set_mon_vals no callback set 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f61b0006110 con 0x7f61a0047a20 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61bd7fa640 1 -- 172.21.15.94:0/849300710 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 1444633209) 0x7f61b0047060 con 0x7f61a0047a20 2024-09-16T10:29:37.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:29:37.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.135+0000 7f61bd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:37.142 INFO:teuthology.orchestra.run.smithi094.stdout:47244640260 2024-09-16T10:29:37.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.137+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.152:6800/2686360781 conn(0x7f61a0039090 legacy=0x7f61a003b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.137+0000 7f61c69bc640 10 monclient: shutdown 2024-09-16T10:29:37.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.137+0000 7f61c69bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:37.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.137+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 >> v1:172.21.15.94:6791/0 conn(0x7f61a0047a20 legacy=0x7f61a0049ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.137+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 shutdown_connections 2024-09-16T10:29:37.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.137+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 >> 172.21.15.94:0/849300710 conn(0x7f61c00a9460 msgr2=0x7f61c00a1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:37.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.137+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 shutdown_connections 2024-09-16T10:29:37.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.137+0000 7f61c69bc640 1 -- 172.21.15.94:0/849300710 wait complete. 2024-09-16T10:29:37.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.5 2024-09-16T10:29:37.150 DEBUG:teuthology.parallel:result is None 2024-09-16T10:29:37.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T10:29:37.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:37.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.244+0000 7f3414df4640 10 monclient: get_monmap_and_config 2024-09-16T10:29:37.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.244+0000 7f3414df4640 10 monclient: build_initial_monmap 2024-09-16T10:29:37.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.244+0000 7f3414df4640 10 monclient: monmap: 2024-09-16T10:29:37.246 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:37.246 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:37.245699+0000 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:37.245699+0000 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:37.247 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:37.248 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:37.248 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.245+0000 7f3414df4640 1 Processor -- start 2024-09-16T10:29:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.245+0000 7f3414df4640 1 -- start start 2024-09-16T10:29:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.245+0000 7f3414df4640 10 monclient: init 2024-09-16T10:29:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 10 monclient: _add_conns ranks=[3,4,0,2,7,6,8,5,1] 2024-09-16T10:29:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 10 monclient(hunting): picked mon.noname-g con 0x7f34100a3860 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 10 monclient(hunting): picked mon.noname-c con 0x7f341006c1d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 10 monclient(hunting): picked mon.noname-a con 0x7f34100a4350 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3410061ae0 con 0x7f34100a4350 2024-09-16T10:29:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34100a0600 con 0x7f34100a3860 2024-09-16T10:29:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34100a0b30 con 0x7f341006c1d0 2024-09-16T10:29:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f3414df4640 10 monclient(hunting): authenticate will time out at 744.586218s 2024-09-16T10:29:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f340dd9b640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f341006c1d0 0x7f341013b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43468/0 (socket says 172.21.15.94:43468) 2024-09-16T10:29:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.246+0000 7f340dd9b640 1 -- 172.21.15.94:0/833937296 learned_addr learned my addr 172.21.15.94:0/833937296 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340ed9d640 1 -- 172.21.15.94:0/833937296 >> v1:172.21.15.94:6789/0 conn(0x7f34100a4350 legacy=0x7f341013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340dd9b640 1 -- 172.21.15.94:0/833937296 >> v1:172.21.15.94:6791/0 conn(0x7f341006c1d0 legacy=0x7f341013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249349194 0 0) 0x7f3410061ae0 con 0x7f34100a4350 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient(hunting): my global_id is 4383 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f8003520 con 0x7f34100a4350 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368285440 0 0) 0x7f34100a0b30 con 0x7f341006c1d0 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient(hunting): my global_id is 4342 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3410061ae0 con 0x7f341006c1d0 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340e59c640 1 -- 172.21.15.94:0/833937296 >> v1:172.21.15.152:6790/0 conn(0x7f34100a3860 legacy=0x7f34100a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2140386879 0 0) 0x7f33f8003520 con 0x7f34100a4350 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 >> v1:172.21.15.94:6791/0 conn(0x7f341006c1d0 legacy=0x7f341013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 >> v1:172.21.15.152:6790/0 conn(0x7f34100a3860 legacy=0x7f34100a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient: _finish_hunting 0 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 monclient: found mon.noname-a 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100a0e10 con 0x7f34100a4350 2024-09-16T10:29:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient: _finish_auth 0 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient: _check_auth_tickets 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3408001aa0 con 0x7f34100a4350 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f3414df4640 5 monclient: authenticate success, global_id 4383 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f3414df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient: dump: 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f3414df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3408003610 con 0x7f34100a4350 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:37.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f3414df4640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 1 -- 172.21.15.94:0/833937296 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3408003ac0 con 0x7f34100a4350 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.248+0000 7f340d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f3414df4640 4 set_mon_vals no callback set 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f340d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f340d59a640 10 monclient: dump: 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f3414df4640 10 monclient: shutdown 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f3414df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f3414df4640 1 -- 172.21.15.94:0/833937296 >> v1:172.21.15.94:6789/0 conn(0x7f34100a4350 legacy=0x7f341013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f3414df4640 1 -- 172.21.15.94:0/833937296 shutdown_connections 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f3414df4640 1 -- 172.21.15.94:0/833937296 >> 172.21.15.94:0/833937296 conn(0x7f34100a5960 msgr2=0x7f34100a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f3414df4640 1 -- 172.21.15.94:0/833937296 shutdown_connections 2024-09-16T10:29:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.249+0000 7f3414df4640 1 -- 172.21.15.94:0/833937296 wait complete. 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.250+0000 7f3414df4640 10 monclient: build_initial_monmap 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.250+0000 7f3414df4640 10 monclient: monmap: 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:37.260 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.250+0000 7f3414df4640 1 Processor -- start 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.250+0000 7f3414df4640 1 -- start start 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.250+0000 7f3414df4640 10 monclient: init 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 10 monclient: _add_conns ranks=[8,3,6,5,1,0,4,7,2] 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 10 monclient(hunting): picked mon.noname-e con 0x7f341006c1d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 10 monclient(hunting): picked mon.noname-g con 0x7f34100a3860 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 10 monclient(hunting): picked mon.noname-d con 0x7f34100a4350 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341006b9a0 con 0x7f34100a3860 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341009f5a0 con 0x7f34100a4350 2024-09-16T10:29:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34101aa920 con 0x7f341006c1d0 2024-09-16T10:29:37.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:37.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f3414df4640 10 monclient(hunting): authenticate will time out at 744.591127s 2024-09-16T10:29:37.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f340e59c640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f341006c1d0 0x7f34101be4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33558/0 (socket says 172.21.15.94:33558) 2024-09-16T10:29:37.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.251+0000 7f340e59c640 1 -- 172.21.15.94:0/4081195955 learned_addr learned my addr 172.21.15.94:0/4081195955 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:37.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f340ed9d640 1 -- 172.21.15.94:0/4081195955 >> v1:172.21.15.94:6792/0 conn(0x7f34100a4350 legacy=0x7f34101d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f340e59c640 1 -- 172.21.15.94:0/4081195955 >> v1:172.21.15.94:6793/0 conn(0x7f341006c1d0 legacy=0x7f34101be4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f340dd9b640 1 -- 172.21.15.94:0/4081195955 >> v1:172.21.15.152:6790/0 conn(0x7f34100a3860 legacy=0x7f34101cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808001657 0 0) 0x7f34100a0e10 con 0x7f34100a4350 2024-09-16T10:29:37.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): my global_id is 4317 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f0003520 con 0x7f34100a4350 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524203361 0 0) 0x7f34101aa920 con 0x7f341006c1d0 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): my global_id is 4319 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34100a0e10 con 0x7f341006c1d0 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183970575 0 0) 0x7f341006b9a0 con 0x7f34100a3860 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): my global_id is 4404 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34101aa920 con 0x7f34100a3860 2024-09-16T10:29:37.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1893821326 0 0) 0x7f34100a0e10 con 0x7f341006c1d0 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f340e59c640 1 -- 172.21.15.94:0/4081195955 >> v1:172.21.15.94:6793/0 conn(0x7f341006c1d0 legacy=0x7f34101be4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f340e59c640 1 -- 172.21.15.94:0/4081195955 >> v1:172.21.15.94:6793/0 conn(0x7f341006c1d0 legacy=0x7f34101be4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f340e59c640 1 --1- 172.21.15.94:0/4081195955 >> v1:172.21.15.94:6793/0 conn(0x7f341006c1d0 0x7f34101be4d0 :-1 s=OPENED pgs=62 cs=1 l=1).handle_message read tag failed 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f340e59c640 1 --1- 172.21.15.94:0/4081195955 >> v1:172.21.15.94:6793/0 conn(0x7f341006c1d0 0x7f34101be4d0 :-1 s=OPENED pgs=62 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f341006b9a0 con 0x7f341006c1d0 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.94:6793/0 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154592053 0 0) 0x7f33f0003520 con 0x7f34100a4350 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34100a0e10 con 0x7f34100a4350 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3408003ac0 con 0x7f34100a4350 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): dump: 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.267 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.268 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1060278901 0 0) 0x7f34101aa920 con 0x7f34100a3860 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33f0003520 con 0x7f34100a3860 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3404001320 con 0x7f34100a3860 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.253+0000 7f33feffd640 10 monclient(hunting): dump: 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1934860368 0 0) 0x7f34100a0e10 con 0x7f34100a4350 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 >> v1:172.21.15.94:6793/0 conn(0x7f341006c1d0 legacy=0x7f34101be4d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 >> v1:172.21.15.152:6790/0 conn(0x7f34100a3860 legacy=0x7f34101cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:37.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 1 monclient: found mon.d 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341009f070 con 0x7f34100a4350 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 5 monclient: authenticate success, global_id 4317 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 10 monclient: _renew_subs 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34101abe90 con 0x7f34100a4350 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 10 monclient: _renew_subs 2024-09-16T10:29:37.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34101bebe0 con 0x7f34100a4350 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3408001ac0 con 0x7f34100a4350 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3408004310 con 0x7f34100a4350 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 10 monclient: dump: 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f340cd99640 4 set_mon_vals no callback set 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f34080106c0 con 0x7f34100a4350 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:37.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:37.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.254+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34101bee30 con 0x7f34100a4350 2024-09-16T10:29:37.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.255+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f3408045cb0 con 0x7f34100a4350 2024-09-16T10:29:37.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.255+0000 7f340e59c640 10 monclient: get_auth_request con 0x7f33f0039aa0 auth_method 0 2024-09-16T10:29:37.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.259+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f34080151f0 con 0x7f34100a4350 2024-09-16T10:29:37.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.259+0000 7f33feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:37.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.259+0000 7f33feffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:37.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.415+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34100a4920 con 0x7f33f0039aa0 2024-09-16T10:29:37.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.417+0000 7f33feffd640 1 -- 172.21.15.94:0/4081195955 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40964 (unknown 2965378022 0 1371881545) 0x7f3400007b60 con 0x7f33f0039aa0 2024-09-16T10:29:37.419 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:37.420 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-09-16T10:29:37.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.419+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 >> v1:172.21.15.152:6800/2686360781 conn(0x7f33f0039aa0 legacy=0x7f33f003bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.419+0000 7f3414df4640 10 monclient: shutdown 2024-09-16T10:29:37.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.419+0000 7f3414df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:37.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.419+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 >> v1:172.21.15.94:6792/0 conn(0x7f34100a4350 legacy=0x7f34101d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.419+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 shutdown_connections 2024-09-16T10:29:37.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.419+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 >> 172.21.15.94:0/4081195955 conn(0x7f34100a5960 msgr2=0x7f341009ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:37.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.419+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 shutdown_connections 2024-09-16T10:29:37.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.419+0000 7f3414df4640 1 -- 172.21.15.94:0/4081195955 wait complete. 2024-09-16T10:29:37.431 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-16T10:29:37.005707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176424,"kb_used_data":1864,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054616,"statfs":{"total":579820584960,"available":579639926784,"internally_reserved":0,"allocated":1908736,"data_stored":1316716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.222963"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967479+0000","last_change":"2024-09-16T10:29:32.967697+0000","last_active":"2024-09-16T10:29:32.967479+0000","last_peered":"2024-09-16T10:29:32.967479+0000","last_clean":"2024-09-16T10:29:32.967479+0000","last_became_active":"2024-09-16T10:29:30.052739+0000","last_became_peered":"2024-09-16T10:29:30.052739+0000","last_unstale":"2024-09-16T10:29:32.967479+0000","last_undegraded":"2024-09-16T10:29:32.967479+0000","last_fullsized":"2024-09-16T10:29:32.967479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:38:23.796765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035088200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.789105+0000","last_change":"2024-09-16T10:29:32.789258+0000","last_active":"2024-09-16T10:29:32.789105+0000","last_peered":"2024-09-16T10:29:32.789105+0000","last_clean":"2024-09-16T10:29:32.789105+0000","last_became_active":"2024-09-16T10:29:30.043984+0000","last_became_peered":"2024-09-16T10:29:30.043984+0000","last_unstale":"2024-09-16T10:29:32.789105+0000","last_undegraded":"2024-09-16T10:29:32.789105+0000","last_fullsized":"2024-09-16T10:29:32.789105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:42.075436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028018599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967691+0000","last_change":"2024-09-16T10:29:32.967858+0000","last_active":"2024-09-16T10:29:32.967691+0000","last_peered":"2024-09-16T10:29:32.967691+0000","last_clean":"2024-09-16T10:29:32.967691+0000","last_became_active":"2024-09-16T10:29:30.043883+0000","last_became_peered":"2024-09-16T10:29:30.043883+0000","last_unstale":"2024-09-16T10:29:32.967691+0000","last_undegraded":"2024-09-16T10:29:32.967691+0000","last_fullsized":"2024-09-16T10:29:32.967691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:31.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048663500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.789054+0000","last_change":"2024-09-16T10:29:32.789216+0000","last_active":"2024-09-16T10:29:32.789054+0000","last_peered":"2024-09-16T10:29:32.789054+0000","last_clean":"2024-09-16T10:29:32.789054+0000","last_became_active":"2024-09-16T10:29:30.039372+0000","last_became_peered":"2024-09-16T10:29:30.039372+0000","last_unstale":"2024-09-16T10:29:32.789054+0000","last_undegraded":"2024-09-16T10:29:32.789054+0000","last_fullsized":"2024-09-16T10:29:32.789054+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:15.482545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027726900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.795029+0000","last_change":"2024-09-16T10:29:32.794417+0000","last_active":"2024-09-16T10:29:32.795029+0000","last_peered":"2024-09-16T10:29:32.795029+0000","last_clean":"2024-09-16T10:29:32.795029+0000","last_became_active":"2024-09-16T10:29:30.039308+0000","last_became_peered":"2024-09-16T10:29:30.039308+0000","last_unstale":"2024-09-16T10:29:32.795029+0000","last_undegraded":"2024-09-16T10:29:32.795029+0000","last_fullsized":"2024-09-16T10:29:32.795029+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:04.076079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076855000000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.784986+0000","last_change":"2024-09-16T10:29:32.784986+0000","last_active":"2024-09-16T10:29:32.784986+0000","last_peered":"2024-09-16T10:29:32.784986+0000","last_clean":"2024-09-16T10:29:32.784986+0000","last_became_active":"2024-09-16T10:29:30.039431+0000","last_became_peered":"2024-09-16T10:29:30.039431+0000","last_unstale":"2024-09-16T10:29:32.784986+0000","last_undegraded":"2024-09-16T10:29:32.784986+0000","last_fullsized":"2024-09-16T10:29:32.784986+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:20.175515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041305900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967435+0000","last_change":"2024-09-16T10:29:32.967651+0000","last_active":"2024-09-16T10:29:32.967435+0000","last_peered":"2024-09-16T10:29:32.967435+0000","last_clean":"2024-09-16T10:29:32.967435+0000","last_became_active":"2024-09-16T10:29:30.043748+0000","last_became_peered":"2024-09-16T10:29:30.043748+0000","last_unstale":"2024-09-16T10:29:32.967435+0000","last_undegraded":"2024-09-16T10:29:32.967435+0000","last_fullsized":"2024-09-16T10:29:32.967435+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:37:47.716642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349687,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.794058+0000","last_change":"2024-09-16T10:29:32.794058+0000","last_active":"2024-09-16T10:29:32.794058+0000","last_peered":"2024-09-16T10:29:32.794058+0000","last_clean":"2024-09-16T10:29:32.794058+0000","last_became_active":"2024-09-16T10:29:30.038374+0000","last_became_peered":"2024-09-16T10:29:30.038374+0000","last_unstale":"2024-09-16T10:29:32.794058+0000","last_undegraded":"2024-09-16T10:29:32.794058+0000","last_fullsized":"2024-09-16T10:29:32.794058+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:17.735950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040381400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.420880+0000","last_change":"2024-09-16T10:29:29.005269+0000","last_active":"2024-09-16T10:29:32.420880+0000","last_peered":"2024-09-16T10:29:32.420880+0000","last_clean":"2024-09-16T10:29:32.420880+0000","last_became_active":"2024-09-16T10:29:29.004894+0000","last_became_peered":"2024-09-16T10:29:29.004894+0000","last_unstale":"2024-09-16T10:29:32.420880+0000","last_undegraded":"2024-09-16T10:29:32.420880+0000","last_fullsized":"2024-09-16T10:29:32.420880+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.994140+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:40.518658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":43156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31256,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340584,"statfs":{"total":96636764160,"available":96604758016,"internally_reserved":0,"allocated":139264,"data_stored":43137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":572065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31264,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340576,"statfs":{"total":96636764160,"available":96604749824,"internally_reserved":0,"allocated":143360,"data_stored":43156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":671744,"data_stored":572065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:37.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.527+0000 7f5fe7bd4640 10 monclient: get_monmap_and_config 2024-09-16T10:29:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.527+0000 7f5fe7bd4640 10 monclient: build_initial_monmap 2024-09-16T10:29:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.527+0000 7f5fe7bd4640 10 monclient: monmap: 2024-09-16T10:29:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:37.528614+0000 2024-09-16T10:29:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:37.528614+0000 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:37.530 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.528+0000 7f5fe7bd4640 1 Processor -- start 2024-09-16T10:29:37.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.528+0000 7f5fe7bd4640 1 -- start start 2024-09-16T10:29:37.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.528+0000 7f5fe7bd4640 10 monclient: init 2024-09-16T10:29:37.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:37.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 10 monclient: _add_conns ranks=[1,8,4,3,7,2,0,5,6] 2024-09-16T10:29:37.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 10 monclient(hunting): picked mon.noname-f con 0x7f5fe00a3860 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:37.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 10 monclient(hunting): picked mon.noname-e con 0x7f5fe006c1d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:37.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 10 monclient(hunting): picked mon.noname-c con 0x7f5fe00a4350 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:37.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe0061ae0 con 0x7f5fe00a3860 2024-09-16T10:29:37.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00a0600 con 0x7f5fe00a4350 2024-09-16T10:29:37.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00a0b30 con 0x7f5fe006c1d0 2024-09-16T10:29:37.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:37.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe7bd4640 10 monclient(hunting): authenticate will time out at 744.869135s 2024-09-16T10:29:37.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe6171640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 0x7f5fe013ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43482/0 (socket says 172.21.15.94:43482) 2024-09-16T10:29:37.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe516f640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f5fe006c1d0 0x7f5fe013b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33562/0 (socket says 172.21.15.94:33562) 2024-09-16T10:29:37.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.529+0000 7f5fe6171640 1 -- 172.21.15.94:0/2160868292 learned_addr learned my addr 172.21.15.94:0/2160868292 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe6171640 1 -- 172.21.15.94:0/2160868292 >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 legacy=0x7f5fe013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe6171640 1 -- 172.21.15.94:0/2160868292 >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 legacy=0x7f5fe013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe6171640 1 --1- 172.21.15.94:0/2160868292 >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 0x7f5fe013ef90 :-1 s=OPENED pgs=69 cs=1 l=1).write_message error sending 0x7f5fe00a0600, (32) Broken pipe 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe516f640 1 -- 172.21.15.94:0/2160868292 >> v1:172.21.15.94:6793/0 conn(0x7f5fe006c1d0 legacy=0x7f5fe013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe6171640 1 --1- 172.21.15.94:0/2160868292 >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 0x7f5fe013ef90 :-1 s=OPENED pgs=69 cs=1 l=1).write_event send msg failed 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe6171640 1 --1- 172.21.15.94:0/2160868292 >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 0x7f5fe013ef90 :-1 s=OPENED pgs=69 cs=1 l=1).write_event send msg failed 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe6171640 1 --1- 172.21.15.94:0/2160868292 >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 0x7f5fe013ef90 :-1 s=OPENED pgs=69 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.94:6791/0 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 -- 172.21.15.94:0/2160868292 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768235828 0 0) 0x7f5fe00a0b30 con 0x7f5fe006c1d0 2024-09-16T10:29:37.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient(hunting): my global_id is 4328 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 -- 172.21.15.94:0/2160868292 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fcc003520 con 0x7f5fe006c1d0 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe5970640 1 -- 172.21.15.94:0/2160868292 >> v1:172.21.15.152:6789/0 conn(0x7f5fe00a3860 legacy=0x7f5fe00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 -- 172.21.15.94:0/2160868292 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3311515904 0 0) 0x7f5fcc003520 con 0x7f5fe006c1d0 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 -- 172.21.15.94:0/2160868292 >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 legacy=0x7f5fe013ef90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 -- 172.21.15.94:0/2160868292 >> v1:172.21.15.152:6789/0 conn(0x7f5fe00a3860 legacy=0x7f5fe00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: _finish_hunting 0 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 monclient: found mon.noname-e 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:37.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 -- 172.21.15.94:0/2160868292 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe00a0e10 con 0x7f5fe006c1d0 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: _finish_auth 0 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: _check_auth_tickets 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 -- 172.21.15.94:0/2160868292 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5fd0001b20 con 0x7f5fe006c1d0 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe7bd4640 5 monclient: authenticate success, global_id 4328 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe7bd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: dump: 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.546 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.547 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe7bd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 -- 172.21.15.94:0/2160868292 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fd00047e0 con 0x7f5fe006c1d0 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 1 -- 172.21.15.94:0/2160868292 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5fd0004c90 con 0x7f5fe006c1d0 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe7bd4640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe7bd4640 4 set_mon_vals no callback set 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.531+0000 7f5fe496e640 10 monclient: dump: 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.532+0000 7f5fe7bd4640 10 monclient: shutdown 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.532+0000 7f5fe7bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.532+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/2160868292 >> v1:172.21.15.94:6793/0 conn(0x7f5fe006c1d0 legacy=0x7f5fe013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.532+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/2160868292 shutdown_connections 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.532+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/2160868292 >> 172.21.15.94:0/2160868292 conn(0x7f5fe00a5960 msgr2=0x7f5fe00a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.532+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/2160868292 shutdown_connections 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.532+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/2160868292 wait complete. 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient: build_initial_monmap 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient: monmap: 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 1 Processor -- start 2024-09-16T10:29:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 1 -- start start 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient: init 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient: _add_conns ranks=[7,6,4,0,1,8,5,2,3] 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient(hunting): picked mon.noname-i con 0x7f5fe006c1d0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient(hunting): picked mon.noname-d con 0x7f5fe00a3860 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient(hunting): picked mon.noname-c con 0x7f5fe00a4350 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe009f650 con 0x7f5fe00a4350 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe006bac0 con 0x7f5fe00a3860 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe01a9cd0 con 0x7f5fe006c1d0 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe7bd4640 10 monclient(hunting): authenticate will time out at 744.873864s 2024-09-16T10:29:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe6171640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 0x7f5fe01d3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43486/0 (socket says 172.21.15.94:43486) 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe516f640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f5fe00a3860 0x7f5fe01cfbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60314/0 (socket says 172.21.15.94:60314) 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.533+0000 7f5fe6171640 1 -- 172.21.15.94:0/3620727432 learned_addr learned my addr 172.21.15.94:0/3620727432 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fe6171640 1 -- 172.21.15.94:0/3620727432 >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 legacy=0x7f5fe01d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fe516f640 1 -- 172.21.15.94:0/3620727432 >> v1:172.21.15.94:6792/0 conn(0x7f5fe00a3860 legacy=0x7f5fe01cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fe5970640 1 -- 172.21.15.94:0/3620727432 >> v1:172.21.15.152:6792/0 conn(0x7f5fe006c1d0 legacy=0x7f5fe01be6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285941122 0 0) 0x7f5fe009f650 con 0x7f5fe00a4350 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): my global_id is 4351 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc4003520 con 0x7f5fe00a4350 2024-09-16T10:29:37.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523019202 0 0) 0x7f5fe006bac0 con 0x7f5fe00a3860 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): my global_id is 4326 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe009f650 con 0x7f5fe00a3860 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 395851900 0 0) 0x7f5fc4003520 con 0x7f5fe00a4350 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe006bac0 con 0x7f5fe00a4350 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3930089992 0 0) 0x7f5fe009f650 con 0x7f5fe00a3860 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fc4003520 con 0x7f5fe00a3860 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5fd8004ea0 con 0x7f5fe00a4350 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): dump: 2024-09-16T10:29:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.555 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804110061 0 0) 0x7f5fe01a9cd0 con 0x7f5fe006c1d0 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): my global_id is 4363 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe009f650 con 0x7f5fe006c1d0 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5fd0001ae0 con 0x7f5fe00a3860 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient(hunting): dump: 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.557 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2172871356 0 0) 0x7f5fc4003520 con 0x7f5fe00a3860 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 >> v1:172.21.15.152:6792/0 conn(0x7f5fe006c1d0 legacy=0x7f5fe01be6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 >> v1:172.21.15.94:6791/0 conn(0x7f5fe00a4350 legacy=0x7f5fe01d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 monclient: found mon.d 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:37.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe009f100 con 0x7f5fe00a3860 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fe7bd4640 5 monclient: authenticate success, global_id 4326 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fe7bd4640 10 monclient: _renew_subs 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fe7bd4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.536+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe01abdc0 con 0x7f5fe00a3860 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fe7bd4640 10 monclient: _renew_subs 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fe7bd4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe01aa5f0 con 0x7f5fe00a3860 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fd0001320 con 0x7f5fe00a3860 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fd67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5fd0004cb0 con 0x7f5fe00a3860 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fd67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fd67fc640 10 monclient: dump: 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.560 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fd7fff640 4 set_mon_vals no callback set 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f5fd0010530 con 0x7f5fe00a3860 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fe7bd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fe7bd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fe7bd4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe01ab800 con 0x7f5fe00a3860 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.537+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f5fd0045e30 con 0x7f5fe00a3860 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.538+0000 7f5fe5970640 10 monclient: get_auth_request con 0x7f5fc40399d0 auth_method 0 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.542+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f5fd0001500 con 0x7f5fe00a3860 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.542+0000 7f5fd67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:37.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.542+0000 7f5fd67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:37.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.700+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5fe00a4920 con 0x7f5fc40399d0 2024-09-16T10:29:37.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.702+0000 7f5fd67fc640 1 -- 172.21.15.94:0/3620727432 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40964 (unknown 2965378022 0 1371881545) 0x7f5fdc0018d0 con 0x7f5fc40399d0 2024-09-16T10:29:37.705 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:37.705 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-09-16T10:29:37.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.704+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 >> v1:172.21.15.152:6800/2686360781 conn(0x7f5fc40399d0 legacy=0x7f5fc403be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.704+0000 7f5fe7bd4640 10 monclient: shutdown 2024-09-16T10:29:37.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.704+0000 7f5fe7bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:37.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.704+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 >> v1:172.21.15.94:6792/0 conn(0x7f5fe00a3860 legacy=0x7f5fe01cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.705+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 shutdown_connections 2024-09-16T10:29:37.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.705+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 >> 172.21.15.94:0/3620727432 conn(0x7f5fe00a5960 msgr2=0x7f5fe009ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:37.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.705+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 shutdown_connections 2024-09-16T10:29:37.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.705+0000 7f5fe7bd4640 1 -- 172.21.15.94:0/3620727432 wait complete. 2024-09-16T10:29:37.716 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-16T10:29:37.005707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176424,"kb_used_data":1864,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054616,"statfs":{"total":579820584960,"available":579639926784,"internally_reserved":0,"allocated":1908736,"data_stored":1316716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.222963"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967479+0000","last_change":"2024-09-16T10:29:32.967697+0000","last_active":"2024-09-16T10:29:32.967479+0000","last_peered":"2024-09-16T10:29:32.967479+0000","last_clean":"2024-09-16T10:29:32.967479+0000","last_became_active":"2024-09-16T10:29:30.052739+0000","last_became_peered":"2024-09-16T10:29:30.052739+0000","last_unstale":"2024-09-16T10:29:32.967479+0000","last_undegraded":"2024-09-16T10:29:32.967479+0000","last_fullsized":"2024-09-16T10:29:32.967479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:38:23.796765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035088200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.789105+0000","last_change":"2024-09-16T10:29:32.789258+0000","last_active":"2024-09-16T10:29:32.789105+0000","last_peered":"2024-09-16T10:29:32.789105+0000","last_clean":"2024-09-16T10:29:32.789105+0000","last_became_active":"2024-09-16T10:29:30.043984+0000","last_became_peered":"2024-09-16T10:29:30.043984+0000","last_unstale":"2024-09-16T10:29:32.789105+0000","last_undegraded":"2024-09-16T10:29:32.789105+0000","last_fullsized":"2024-09-16T10:29:32.789105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:42.075436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028018599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967691+0000","last_change":"2024-09-16T10:29:32.967858+0000","last_active":"2024-09-16T10:29:32.967691+0000","last_peered":"2024-09-16T10:29:32.967691+0000","last_clean":"2024-09-16T10:29:32.967691+0000","last_became_active":"2024-09-16T10:29:30.043883+0000","last_became_peered":"2024-09-16T10:29:30.043883+0000","last_unstale":"2024-09-16T10:29:32.967691+0000","last_undegraded":"2024-09-16T10:29:32.967691+0000","last_fullsized":"2024-09-16T10:29:32.967691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:31.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048663500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.789054+0000","last_change":"2024-09-16T10:29:32.789216+0000","last_active":"2024-09-16T10:29:32.789054+0000","last_peered":"2024-09-16T10:29:32.789054+0000","last_clean":"2024-09-16T10:29:32.789054+0000","last_became_active":"2024-09-16T10:29:30.039372+0000","last_became_peered":"2024-09-16T10:29:30.039372+0000","last_unstale":"2024-09-16T10:29:32.789054+0000","last_undegraded":"2024-09-16T10:29:32.789054+0000","last_fullsized":"2024-09-16T10:29:32.789054+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:15.482545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027726900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.795029+0000","last_change":"2024-09-16T10:29:32.794417+0000","last_active":"2024-09-16T10:29:32.795029+0000","last_peered":"2024-09-16T10:29:32.795029+0000","last_clean":"2024-09-16T10:29:32.795029+0000","last_became_active":"2024-09-16T10:29:30.039308+0000","last_became_peered":"2024-09-16T10:29:30.039308+0000","last_unstale":"2024-09-16T10:29:32.795029+0000","last_undegraded":"2024-09-16T10:29:32.795029+0000","last_fullsized":"2024-09-16T10:29:32.795029+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:04.076079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076855000000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.784986+0000","last_change":"2024-09-16T10:29:32.784986+0000","last_active":"2024-09-16T10:29:32.784986+0000","last_peered":"2024-09-16T10:29:32.784986+0000","last_clean":"2024-09-16T10:29:32.784986+0000","last_became_active":"2024-09-16T10:29:30.039431+0000","last_became_peered":"2024-09-16T10:29:30.039431+0000","last_unstale":"2024-09-16T10:29:32.784986+0000","last_undegraded":"2024-09-16T10:29:32.784986+0000","last_fullsized":"2024-09-16T10:29:32.784986+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:20.175515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041305900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967435+0000","last_change":"2024-09-16T10:29:32.967651+0000","last_active":"2024-09-16T10:29:32.967435+0000","last_peered":"2024-09-16T10:29:32.967435+0000","last_clean":"2024-09-16T10:29:32.967435+0000","last_became_active":"2024-09-16T10:29:30.043748+0000","last_became_peered":"2024-09-16T10:29:30.043748+0000","last_unstale":"2024-09-16T10:29:32.967435+0000","last_undegraded":"2024-09-16T10:29:32.967435+0000","last_fullsized":"2024-09-16T10:29:32.967435+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:37:47.716642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349687,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.794058+0000","last_change":"2024-09-16T10:29:32.794058+0000","last_active":"2024-09-16T10:29:32.794058+0000","last_peered":"2024-09-16T10:29:32.794058+0000","last_clean":"2024-09-16T10:29:32.794058+0000","last_became_active":"2024-09-16T10:29:30.038374+0000","last_became_peered":"2024-09-16T10:29:30.038374+0000","last_unstale":"2024-09-16T10:29:32.794058+0000","last_undegraded":"2024-09-16T10:29:32.794058+0000","last_fullsized":"2024-09-16T10:29:32.794058+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:17.735950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040381400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.420880+0000","last_change":"2024-09-16T10:29:29.005269+0000","last_active":"2024-09-16T10:29:32.420880+0000","last_peered":"2024-09-16T10:29:32.420880+0000","last_clean":"2024-09-16T10:29:32.420880+0000","last_became_active":"2024-09-16T10:29:29.004894+0000","last_became_peered":"2024-09-16T10:29:29.004894+0000","last_unstale":"2024-09-16T10:29:32.420880+0000","last_undegraded":"2024-09-16T10:29:32.420880+0000","last_fullsized":"2024-09-16T10:29:32.420880+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.994140+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:40.518658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":43156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31256,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340584,"statfs":{"total":96636764160,"available":96604758016,"internally_reserved":0,"allocated":139264,"data_stored":43137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":572065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31264,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340576,"statfs":{"total":96636764160,"available":96604749824,"internally_reserved":0,"allocated":143360,"data_stored":43156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":671744,"data_stored":572065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:37.717 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T10:29:37.717 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T10:29:37.717 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T10:29:37.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T10:29:37.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.810+0000 7efc25118640 10 monclient: get_monmap_and_config 2024-09-16T10:29:37.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.810+0000 7efc25118640 10 monclient: build_initial_monmap 2024-09-16T10:29:37.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.810+0000 7efc25118640 10 monclient: monmap: 2024-09-16T10:29:37.812 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:37.811693+0000 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:37.811693+0000 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:37.813 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:37.814 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:37.814 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.811+0000 7efc25118640 1 Processor -- start 2024-09-16T10:29:37.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.811+0000 7efc25118640 1 -- start start 2024-09-16T10:29:37.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.811+0000 7efc25118640 10 monclient: init 2024-09-16T10:29:37.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.811+0000 7efc25118640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:37.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.811+0000 7efc25118640 10 monclient: _add_conns ranks=[5,4,0,8,3,1,7,2,6] 2024-09-16T10:29:37.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.811+0000 7efc25118640 10 monclient(hunting): picked mon.noname-h con 0x7efc200a3840 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:37.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc25118640 10 monclient(hunting): picked mon.noname-c con 0x7efc200a3c20 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:37.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc25118640 10 monclient(hunting): picked mon.noname-a con 0x7efc2013ca60 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:37.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc25118640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc20061ae0 con 0x7efc2013ca60 2024-09-16T10:29:37.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc25118640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc200a52c0 con 0x7efc200a3c20 2024-09-16T10:29:37.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc25118640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2006c470 con 0x7efc200a3840 2024-09-16T10:29:37.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc25118640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:37.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc25118640 10 monclient(hunting): authenticate will time out at 745.152057s 2024-09-16T10:29:37.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc1659c640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7efc200a3c20 0x7efc200a4000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43500/0 (socket says 172.21.15.94:43500) 2024-09-16T10:29:37.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc1f59e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efc2013ca60 0x7efc2013ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41546/0 (socket says 172.21.15.94:41546) 2024-09-16T10:29:37.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.812+0000 7efc1659c640 1 -- 172.21.15.94:0/733918304 learned_addr learned my addr 172.21.15.94:0/733918304 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:37.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1f59e640 1 -- 172.21.15.94:0/733918304 >> v1:172.21.15.94:6789/0 conn(0x7efc2013ca60 legacy=0x7efc2013ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1659c640 1 -- 172.21.15.94:0/733918304 >> v1:172.21.15.94:6791/0 conn(0x7efc200a3c20 legacy=0x7efc200a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53032699 0 0) 0x7efc20061ae0 con 0x7efc2013ca60 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient(hunting): my global_id is 4392 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc0c003520 con 0x7efc2013ca60 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539638878 0 0) 0x7efc200a52c0 con 0x7efc200a3c20 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient(hunting): my global_id is 4360 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc20061ae0 con 0x7efc200a3c20 2024-09-16T10:29:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1ed9d640 1 -- 172.21.15.94:0/733918304 >> v1:172.21.15.152:6791/0 conn(0x7efc200a3840 legacy=0x7efc200a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3113605352 0 0) 0x7efc20061ae0 con 0x7efc200a3c20 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 >> v1:172.21.15.152:6791/0 conn(0x7efc200a3840 legacy=0x7efc200a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 >> v1:172.21.15.94:6789/0 conn(0x7efc2013ca60 legacy=0x7efc2013ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: _finish_hunting 0 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 monclient: found mon.noname-c 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2006c6b0 con 0x7efc200a3c20 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: _finish_auth 0 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: _check_auth_tickets 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efc08001a60 con 0x7efc200a3c20 2024-09-16T10:29:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc25118640 5 monclient: authenticate success, global_id 4360 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc25118640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: dump: 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc25118640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc08001d70 con 0x7efc200a3c20 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 1 -- 172.21.15.94:0/733918304 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efc08004be0 con 0x7efc200a3c20 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc25118640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc25118640 4 set_mon_vals no callback set 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.814+0000 7efc1e59c640 10 monclient: dump: 2024-09-16T10:29:37.823 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.815+0000 7efc25118640 10 monclient: shutdown 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.815+0000 7efc25118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.815+0000 7efc25118640 1 -- 172.21.15.94:0/733918304 >> v1:172.21.15.94:6791/0 conn(0x7efc200a3c20 legacy=0x7efc200a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.815+0000 7efc25118640 1 -- 172.21.15.94:0/733918304 shutdown_connections 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.815+0000 7efc25118640 1 -- 172.21.15.94:0/733918304 >> 172.21.15.94:0/733918304 conn(0x7efc200a9530 msgr2=0x7efc200a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.815+0000 7efc25118640 1 -- 172.21.15.94:0/733918304 shutdown_connections 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.815+0000 7efc25118640 1 -- 172.21.15.94:0/733918304 wait complete. 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.816+0000 7efc25118640 10 monclient: build_initial_monmap 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.816+0000 7efc25118640 10 monclient: monmap: 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.816+0000 7efc25118640 1 Processor -- start 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.816+0000 7efc25118640 1 -- start start 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.816+0000 7efc25118640 10 monclient: init 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 10 monclient: _add_conns ranks=[4,6,3,5,7,0,2,8,1] 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 10 monclient(hunting): picked mon.noname-c con 0x7efc200a3840 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 10 monclient(hunting): picked mon.noname-d con 0x7efc200a3c20 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 10 monclient(hunting): picked mon.noname-g con 0x7efc2013ca60 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc20123ce0 con 0x7efc2013ca60 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2013c0f0 con 0x7efc200a3840 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc20125190 con 0x7efc200a3c20 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc25118640 10 monclient(hunting): authenticate will time out at 745.157106s 2024-09-16T10:29:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc1ed9d640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7efc200a3840 0x7efc201caec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43510/0 (socket says 172.21.15.94:43510) 2024-09-16T10:29:37.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.817+0000 7efc1ed9d640 1 -- 172.21.15.94:0/2105130630 learned_addr learned my addr 172.21.15.94:0/2105130630 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1659c640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6792/0 conn(0x7efc200a3c20 legacy=0x7efc201cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1ed9d640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6791/0 conn(0x7efc200a3840 legacy=0x7efc201caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1659c640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6792/0 conn(0x7efc200a3c20 legacy=0x7efc201cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:29:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1659c640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6792/0 conn(0x7efc200a3c20 legacy=0x7efc201cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:29:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1659c640 1 --1- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6792/0 conn(0x7efc200a3c20 0x7efc201cff00 :-1 s=OPENED pgs=78 cs=1 l=1).handle_message read tag failed 2024-09-16T10:29:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1659c640 1 --1- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6792/0 conn(0x7efc200a3c20 0x7efc201cff00 :-1 s=OPENED pgs=78 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:29:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1f59e640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.152:6790/0 conn(0x7efc2013ca60 legacy=0x7efc201d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.94:6792/0 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519528854 0 0) 0x7efc2013c0f0 con 0x7efc200a3840 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): my global_id is 4369 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc00003550 con 0x7efc200a3840 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1ed9d640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6791/0 conn(0x7efc200a3840 legacy=0x7efc201caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1ed9d640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6791/0 conn(0x7efc200a3840 legacy=0x7efc201caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1ed9d640 1 --1- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6791/0 conn(0x7efc200a3840 0x7efc201caec0 :-1 s=OPENED pgs=72 cs=1 l=1).handle_message read tag failed 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc1ed9d640 1 --1- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6791/0 conn(0x7efc200a3840 0x7efc201caec0 :-1 s=OPENED pgs=72 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.94:6791/0 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653642106 0 0) 0x7efc20123ce0 con 0x7efc2013ca60 2024-09-16T10:29:37.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): my global_id is 4413 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2013c0f0 con 0x7efc2013ca60 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3747762360 0 0) 0x7efc2013c0f0 con 0x7efc2013ca60 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc20123ce0 con 0x7efc2013ca60 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efc18001ff0 con 0x7efc2013ca60 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): dump: 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.834 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.819+0000 7efc17fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2664732738 0 0) 0x7efc20123ce0 con 0x7efc2013ca60 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6792/0 conn(0x7efc200a3c20 legacy=0x7efc201cff00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.94:6791/0 conn(0x7efc200a3840 legacy=0x7efc201caec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 1 monclient: found mon.g 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc201230a0 con 0x7efc2013ca60 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 10 monclient: _finish_auth 0 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 5 monclient: authenticate success, global_id 4413 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 10 monclient: _renew_subs 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc20124eb0 con 0x7efc2013ca60 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 10 monclient: _renew_subs 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc201239f0 con 0x7efc2013ca60 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc18001c60 con 0x7efc2013ca60 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efc18005490 con 0x7efc2013ca60 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc1dd9b640 4 set_mon_vals no callback set 2024-09-16T10:29:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 10 monclient: dump: 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7efc180104f0 con 0x7efc2013ca60 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.820+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 --> v1:172.21.15.152:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc200a7310 con 0x7efc2013ca60 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.821+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7efc18046190 con 0x7efc2013ca60 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.821+0000 7efc1ed9d640 10 monclient: get_auth_request con 0x7efc00039060 auth_method 0 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.826+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7efc18015670 con 0x7efc2013ca60 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.826+0000 7efc17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:37.826+0000 7efc17fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:38.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.023+0000 7efc25118640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T10:29:38.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.023+0000 7efc25118640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T10:29:38.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.023+0000 7efc25118640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:38.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.023+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 --> v1:172.21.15.152:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc20066570 con 0x7efc2013ca60 2024-09-16T10:29:38.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.023+0000 7efc17fff640 1 -- 172.21.15.94:0/2105130630 <== mon.3 v1:172.21.15.152:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7efc18013ae0 con 0x7efc2013ca60 2024-09-16T10:29:38.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.023+0000 7efc17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T10:29:38.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.023+0000 7efc17fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:38.026 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T10:29:38.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.026+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.152:6800/2686360781 conn(0x7efc00039060 legacy=0x7efc0003b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.026+0000 7efc25118640 10 monclient: shutdown 2024-09-16T10:29:38.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.026+0000 7efc25118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:38.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.026+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 >> v1:172.21.15.152:6790/0 conn(0x7efc2013ca60 legacy=0x7efc201d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.027+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 shutdown_connections 2024-09-16T10:29:38.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.027+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 >> 172.21.15.94:0/2105130630 conn(0x7efc200a9530 msgr2=0x7efc200a0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:38.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.027+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 shutdown_connections 2024-09-16T10:29:38.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.027+0000 7efc25118640 1 -- 172.21.15.94:0/2105130630 wait complete. 2024-09-16T10:29:38.039 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T10:29:38.039 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T10:29:38.054 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T10:29:38.054 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:29:38.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.149+0000 7f6fe91e2640 10 monclient: get_monmap_and_config 2024-09-16T10:29:38.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.149+0000 7f6fe91e2640 10 monclient: build_initial_monmap 2024-09-16T10:29:38.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.149+0000 7f6fe91e2640 10 monclient: monmap: 2024-09-16T10:29:38.150 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:38.149818+0000 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:38.149818+0000 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:38.151 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:38.152 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:38.152 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:38.152 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.149+0000 7f6fe91e2640 1 Processor -- start 2024-09-16T10:29:38.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.149+0000 7f6fe91e2640 1 -- start start 2024-09-16T10:29:38.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.149+0000 7f6fe91e2640 10 monclient: init 2024-09-16T10:29:38.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:38.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 10 monclient: _add_conns ranks=[8,4,5,7,6,1,0,2,3] 2024-09-16T10:29:38.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 10 monclient(hunting): picked mon.noname-e con 0x7f6fe40a3860 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:38.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe406c1d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:38.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 10 monclient(hunting): picked mon.noname-h con 0x7f6fe40a4350 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:38.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe4061ae0 con 0x7f6fe406c1d0 2024-09-16T10:29:38.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe40a0600 con 0x7f6fe40a4350 2024-09-16T10:29:38.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe40a0b30 con 0x7f6fe40a3860 2024-09-16T10:29:38.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:38.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe91e2640 10 monclient(hunting): authenticate will time out at 745.490206s 2024-09-16T10:29:38.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe259c640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f6fe406c1d0 0x7f6fe413b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43522/0 (socket says 172.21.15.94:43522) 2024-09-16T10:29:38.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.150+0000 7f6fe259c640 1 -- 172.21.15.94:0/2974890401 learned_addr learned my addr 172.21.15.94:0/2974890401 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe259c640 1 -- 172.21.15.94:0/2974890401 >> v1:172.21.15.94:6791/0 conn(0x7f6fe406c1d0 legacy=0x7f6fe413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe2d9d640 1 -- 172.21.15.94:0/2974890401 >> v1:172.21.15.94:6793/0 conn(0x7f6fe40a3860 legacy=0x7f6fe40a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606221574 0 0) 0x7f6fe4061ae0 con 0x7f6fe406c1d0 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): my global_id is 4378 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe359e640 1 -- 172.21.15.94:0/2974890401 >> v1:172.21.15.152:6791/0 conn(0x7f6fe40a4350 legacy=0x7f6fe413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc003520 con 0x7f6fe406c1d0 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982748287 0 0) 0x7f6fe40a0b30 con 0x7f6fe40a3860 2024-09-16T10:29:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): my global_id is 4337 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe4061ae0 con 0x7f6fe40a3860 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605627423 0 0) 0x7f6fe40a0600 con 0x7f6fe40a4350 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.152+0000 7f6fe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient(hunting): my global_id is 4370 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe40a0b30 con 0x7f6fe40a4350 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558977963 0 0) 0x7f6fe4061ae0 con 0x7f6fe40a3860 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 >> v1:172.21.15.152:6791/0 conn(0x7f6fe40a4350 legacy=0x7f6fe413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 >> v1:172.21.15.94:6791/0 conn(0x7f6fe406c1d0 legacy=0x7f6fe413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: _finish_hunting 0 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 1 monclient: found mon.noname-e 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40a0e10 con 0x7f6fe40a3860 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: _finish_auth 0 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: _check_auth_tickets 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6fd4001b70 con 0x7f6fe40a3860 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 5 monclient: authenticate success, global_id 4337 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: dump: 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.161 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fd4003610 con 0x7f6fe40a3860 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 1 -- 172.21.15.94:0/2974890401 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6fd4003ac0 con 0x7f6fe40a3860 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 4 set_mon_vals no callback set 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe1d9b640 10 monclient: dump: 2024-09-16T10:29:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 10 monclient: shutdown 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 1 -- 172.21.15.94:0/2974890401 >> v1:172.21.15.94:6793/0 conn(0x7f6fe40a3860 legacy=0x7f6fe40a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 1 -- 172.21.15.94:0/2974890401 shutdown_connections 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 1 -- 172.21.15.94:0/2974890401 >> 172.21.15.94:0/2974890401 conn(0x7f6fe40a5960 msgr2=0x7f6fe40a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 1 -- 172.21.15.94:0/2974890401 shutdown_connections 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.153+0000 7f6fe91e2640 1 -- 172.21.15.94:0/2974890401 wait complete. 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.154+0000 7f6fe91e2640 10 monclient: build_initial_monmap 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.154+0000 7f6fe91e2640 10 monclient: monmap: 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:38.165 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.154+0000 7f6fe91e2640 1 Processor -- start 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 1 -- start start 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 10 monclient: init 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 10 monclient: _add_conns ranks=[5,4,3,8,1,7,0,6,2] 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 10 monclient(hunting): picked mon.noname-h con 0x7f6fe406c1d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe40a3860 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 10 monclient(hunting): picked mon.noname-g con 0x7f6fe40a4350 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe409f650 con 0x7f6fe40a4350 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe406ba50 con 0x7f6fe40a3860 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe41cb630 con 0x7f6fe406c1d0 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:38.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe91e2640 10 monclient(hunting): authenticate will time out at 745.495537s 2024-09-16T10:29:38.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe259c640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f6fe40a3860 0x7f6fe41cfa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43528/0 (socket says 172.21.15.94:43528) 2024-09-16T10:29:38.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.155+0000 7f6fe259c640 1 -- 172.21.15.94:0/1402906706 learned_addr learned my addr 172.21.15.94:0/1402906706 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:38.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.157+0000 7f6fe259c640 1 -- 172.21.15.94:0/1402906706 >> v1:172.21.15.94:6791/0 conn(0x7f6fe40a3860 legacy=0x7f6fe41cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.157+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37845674 0 0) 0x7f6fe406ba50 con 0x7f6fe40a3860 2024-09-16T10:29:38.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.157+0000 7f6fcb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.157+0000 7f6fcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.157+0000 7f6fcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.157+0000 7f6fcb7fe640 10 monclient(hunting): my global_id is 4387 2024-09-16T10:29:38.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.157+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd0003520 con 0x7f6fe40a3860 2024-09-16T10:29:38.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.157+0000 7f6fe359e640 1 -- 172.21.15.94:0/1402906706 >> v1:172.21.15.152:6790/0 conn(0x7f6fe40a4350 legacy=0x7f6fe41d3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.157+0000 7f6fe2d9d640 1 -- 172.21.15.94:0/1402906706 >> v1:172.21.15.152:6791/0 conn(0x7f6fe406c1d0 legacy=0x7f6fe41be470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3086147027 0 0) 0x7f6fd0003520 con 0x7f6fe40a3860 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe406ba50 con 0x7f6fe40a3860 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6fd8001d80 con 0x7f6fe40a3860 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): dump: 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686518143 0 0) 0x7f6fe409f650 con 0x7f6fe40a4350 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): my global_id is 4422 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd0003520 con 0x7f6fe40a4350 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610136450 0 0) 0x7f6fe41cb630 con 0x7f6fe406c1d0 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient(hunting): my global_id is 4379 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe409f650 con 0x7f6fe406c1d0 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1098125690 0 0) 0x7f6fe406ba50 con 0x7f6fe40a3860 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 >> v1:172.21.15.152:6791/0 conn(0x7f6fe406c1d0 legacy=0x7f6fe41be470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 >> v1:172.21.15.152:6790/0 conn(0x7f6fe40a4350 legacy=0x7f6fe41d3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 monclient: found mon.c 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe41aa8c0 con 0x7f6fe40a3860 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fe91e2640 5 monclient: authenticate success, global_id 4387 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fe91e2640 10 monclient: _renew_subs 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fe91e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe409f100 con 0x7f6fe40a3860 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fe91e2640 10 monclient: _renew_subs 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fe91e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe41ab590 con 0x7f6fe40a3860 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fd8004dd0 con 0x7f6fe40a3860 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6fd8005280 con 0x7f6fe40a3860 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fcb7fe640 10 monclient: dump: 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.158+0000 7f6fe159a640 4 set_mon_vals no callback set 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.159+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f6fd80102e0 con 0x7f6fe40a3860 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.159+0000 7f6fe91e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.159+0000 7f6fe91e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:38.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.159+0000 7f6fe91e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.159+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe40a3210 con 0x7f6fe40a3860 2024-09-16T10:29:38.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.159+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f6fd8046000 con 0x7f6fe40a3860 2024-09-16T10:29:38.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.159+0000 7f6fe2d9d640 10 monclient: get_auth_request con 0x7f6fd0039230 auth_method 0 2024-09-16T10:29:38.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.163+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f6fd80154e0 con 0x7f6fe40a3860 2024-09-16T10:29:38.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.163+0000 7f6fcb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:38.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.163+0000 7f6fcb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:38.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.326+0000 7f6fe91e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:38.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.326+0000 7f6fe91e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:38.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.326+0000 7f6fe91e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.326+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fe41a9ad0 con 0x7f6fe40a3860 2024-09-16T10:29:38.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.327+0000 7f6fcb7fe640 1 -- 172.21.15.94:0/1402906706 <== mon.4 v1:172.21.15.94:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10830 (unknown 1054498076 0 24213607) 0x7f6fd80150d0 con 0x7f6fe40a3860 2024-09-16T10:29:38.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.327+0000 7f6fcb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:29:38.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.327+0000 7f6fcb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:38.329 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:38.329 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":16,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","created":"2024-09-16T10:29:16.029056+0000","modified":"2024-09-16T10:29:32.780214+0000","last_up_change":"2024-09-16T10:29:27.994140+0000","last_in_change":"2024-09-16T10:29:20.181320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:29:27.046989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T10:29:28.942998+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"19cab4f7-c271-4ce8-895e-75a5871502f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":4202994153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":4202994153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":4202994153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":4202994153}]},"public_addr":"172.21.15.94:6804/4202994153","cluster_addr":"172.21.15.94:6806/4202994153","heartbeat_back_addr":"172.21.15.94:6810/4202994153","heartbeat_front_addr":"172.21.15.94:6808/4202994153","state":["exists","up"]},{"osd":1,"uuid":"6641fac5-e871-4401-bf30-598b633e70b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":782646670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":782646670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":782646670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":782646670}]},"public_addr":"172.21.15.94:6800/782646670","cluster_addr":"172.21.15.94:6801/782646670","heartbeat_back_addr":"172.21.15.94:6803/782646670","heartbeat_front_addr":"172.21.15.94:6802/782646670","state":["exists","up"]},{"osd":2,"uuid":"6f4cedf2-071d-44f9-87a3-a766668fcd03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3102113414}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3102113414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3102113414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3102113414}]},"public_addr":"172.21.15.94:6805/3102113414","cluster_addr":"172.21.15.94:6807/3102113414","heartbeat_back_addr":"172.21.15.94:6811/3102113414","heartbeat_front_addr":"172.21.15.94:6809/3102113414","state":["exists","up"]},{"osd":3,"uuid":"3634242e-61e0-4221-8c38-7dce4d5053ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6805","nonce":823649059}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6807","nonce":823649059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6811","nonce":823649059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6809","nonce":823649059}]},"public_addr":"172.21.15.152:6805/823649059","cluster_addr":"172.21.15.152:6807/823649059","heartbeat_back_addr":"172.21.15.152:6811/823649059","heartbeat_front_addr":"172.21.15.152:6809/823649059","state":["exists","up"]},{"osd":4,"uuid":"eed750dc-2d29-443f-a91c-d9c5abdcf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6801","nonce":1475699231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6802","nonce":1475699231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6804","nonce":1475699231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6803","nonce":1475699231}]},"public_addr":"172.21.15.152:6801/1475699231","cluster_addr":"172.21.15.152:6802/1475699231","heartbeat_back_addr":"172.21.15.152:6804/1475699231","heartbeat_front_addr":"172.21.15.152:6803/1475699231","state":["exists","up"]},{"osd":5,"uuid":"5d6daac7-1ac7-4f0a-b315-27d1333fc3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6806","nonce":3657457384}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6808","nonce":3657457384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6812","nonce":3657457384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6810","nonce":3657457384}]},"public_addr":"172.21.15.152:6806/3657457384","cluster_addr":"172.21.15.152:6808/3657457384","heartbeat_back_addr":"172.21.15.152:6812/3657457384","heartbeat_front_addr":"172.21.15.152:6810/3657457384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.818679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.874108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.957469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.872051+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.780789+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.901652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:29:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.329+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 >> v1:172.21.15.152:6800/2686360781 conn(0x7f6fd0039230 legacy=0x7f6fd003b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.329+0000 7f6fe91e2640 10 monclient: shutdown 2024-09-16T10:29:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.329+0000 7f6fe91e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.329+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 >> v1:172.21.15.94:6791/0 conn(0x7f6fe40a3860 legacy=0x7f6fe41cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.329+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 shutdown_connections 2024-09-16T10:29:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.329+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 >> 172.21.15.94:0/1402906706 conn(0x7f6fe40a5960 msgr2=0x7f6fe40a77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.329+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 shutdown_connections 2024-09-16T10:29:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.329+0000 7f6fe91e2640 1 -- 172.21.15.94:0/1402906706 wait complete. 2024-09-16T10:29:38.342 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:29:27.046989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T10:29:28.942998+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T10:29:38.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T10:29:38.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.439+0000 7fe502dbc640 10 monclient: get_monmap_and_config 2024-09-16T10:29:38.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.439+0000 7fe502dbc640 10 monclient: build_initial_monmap 2024-09-16T10:29:38.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.439+0000 7fe502dbc640 10 monclient: monmap: 2024-09-16T10:29:38.441 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:38.441 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.441 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:38.440593+0000 2024-09-16T10:29:38.441 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:38.440593+0000 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:38.442 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.440+0000 7fe502dbc640 1 Processor -- start 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.440+0000 7fe502dbc640 1 -- start start 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.440+0000 7fe502dbc640 10 monclient: init 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.440+0000 7fe502dbc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe502dbc640 10 monclient: _add_conns ranks=[5,3,1,8,2,6,7,4,0] 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe502dbc640 10 monclient(hunting): picked mon.noname-h con 0x7fe4fc1368e0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe502dbc640 10 monclient(hunting): picked mon.noname-g con 0x7fe4fc136cc0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe502dbc640 10 monclient(hunting): picked mon.noname-f con 0x7fe4fc1370a0 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe502dbc640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc061ae0 con 0x7fe4fc1370a0 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe502dbc640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc0a03e0 con 0x7fe4fc136cc0 2024-09-16T10:29:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe502dbc640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc06c8d0 con 0x7fe4fc1368e0 2024-09-16T10:29:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe502dbc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe502dbc640 10 monclient(hunting): authenticate will time out at 745.781115s 2024-09-16T10:29:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe500b58640 1 --1- >> v1:172.21.15.152:6791/0 conn(0x7fe4fc1368e0 0x7fe4fc0a1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6791/0 says I am v1:172.21.15.94:45968/0 (socket says 172.21.15.94:45968) 2024-09-16T10:29:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe4f3fff640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7fe4fc136cc0 0x7fe4fc13b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.94:46504/0 (socket says 172.21.15.94:46504) 2024-09-16T10:29:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe500b58640 1 -- 172.21.15.94:0/31971928 learned_addr learned my addr 172.21.15.94:0/31971928 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.441+0000 7fe501359640 1 --1- 172.21.15.94:0/31971928 >> v1:172.21.15.152:6789/0 conn(0x7fe4fc1370a0 0x7fe4fc13ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.94:49730/0 (socket says 172.21.15.94:49730) 2024-09-16T10:29:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe501359640 1 -- 172.21.15.94:0/31971928 >> v1:172.21.15.152:6789/0 conn(0x7fe4fc1370a0 legacy=0x7fe4fc13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f3fff640 1 -- 172.21.15.94:0/31971928 >> v1:172.21.15.152:6790/0 conn(0x7fe4fc136cc0 legacy=0x7fe4fc13b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe500b58640 1 -- 172.21.15.94:0/31971928 >> v1:172.21.15.152:6791/0 conn(0x7fe4fc1368e0 legacy=0x7fe4fc0a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068500729 0 0) 0x7fe4fc061ae0 con 0x7fe4fc1370a0 2024-09-16T10:29:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): my global_id is 4348 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e4003520 con 0x7fe4fc1370a0 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437346593 0 0) 0x7fe4fc0a03e0 con 0x7fe4fc136cc0 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): my global_id is 4431 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc061ae0 con 0x7fe4fc136cc0 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533425951 0 0) 0x7fe4fc06c8d0 con 0x7fe4fc1368e0 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient(hunting): my global_id is 4388 2024-09-16T10:29:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc0a03e0 con 0x7fe4fc1368e0 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2698505870 0 0) 0x7fe4e4003520 con 0x7fe4fc1370a0 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 >> v1:172.21.15.152:6791/0 conn(0x7fe4fc1368e0 legacy=0x7fe4fc0a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 >> v1:172.21.15.152:6790/0 conn(0x7fe4fc136cc0 legacy=0x7fe4fc13b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.443+0000 7fe4f37fe640 1 monclient: found mon.noname-f 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.152:6789/0 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc0a0680 con 0x7fe4fc1370a0 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4f4001ea0 con 0x7fe4fc1370a0 2024-09-16T10:29:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 10 monclient: dump: 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 5 monclient: authenticate success, global_id 4348 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe501359640 1 --1- 172.21.15.94:0/31971928 >> v1:172.21.15.152:6789/0 conn(0x7fe4fc1370a0 0x7fe4fc13ef00 :-1 s=OPENED pgs=64 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7fe4f4004b60 config(0 keys) v1 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 <== mon.1 v1:172.21.15.152:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4f4004b60 con 0x7fe4fc1370a0 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 4 set_mon_vals no callback set 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 10 monclient: shutdown 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 1 -- 172.21.15.94:0/31971928 >> v1:172.21.15.152:6789/0 conn(0x7fe4fc1370a0 legacy=0x7fe4fc13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 1 -- 172.21.15.94:0/31971928 <== mon.1 v1:172.21.15.152:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4f4005010 con 0x7fe4fc1370a0 2024-09-16T10:29:38.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe4f37fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 1 -- 172.21.15.94:0/31971928 shutdown_connections 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 1 -- 172.21.15.94:0/31971928 >> 172.21.15.94:0/31971928 conn(0x7fe4fc0a58d0 msgr2=0x7fe4fc0a5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 1 -- 172.21.15.94:0/31971928 shutdown_connections 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.444+0000 7fe502dbc640 1 -- 172.21.15.94:0/31971928 wait complete. 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.445+0000 7fe502dbc640 10 monclient: build_initial_monmap 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.445+0000 7fe502dbc640 10 monclient: monmap: 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.445+0000 7fe502dbc640 1 Processor -- start 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.445+0000 7fe502dbc640 1 -- start start 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.445+0000 7fe502dbc640 10 monclient: init 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 10 monclient: _add_conns ranks=[2,4,6,7,5,3,0,8,1] 2024-09-16T10:29:38.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 10 monclient(hunting): picked mon.noname-b con 0x7fe4fc1368e0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 10 monclient(hunting): picked mon.noname-c con 0x7fe4fc136cc0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 10 monclient(hunting): picked mon.noname-d con 0x7fe4fc1370a0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc0a8a70 con 0x7fe4fc1368e0 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc0a2a70 con 0x7fe4fc136cc0 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc0a8620 con 0x7fe4fc1370a0 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe502dbc640 10 monclient(hunting): authenticate will time out at 745.786396s 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe500b58640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fe4fc1368e0 0x7fe4fc1359b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48438/0 (socket says 172.21.15.94:48438) 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe4f3fff640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fe4fc136cc0 0x7fe4fc1cfb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43536/0 (socket says 172.21.15.94:43536) 2024-09-16T10:29:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.446+0000 7fe500b58640 1 -- 172.21.15.94:0/3009876967 learned_addr learned my addr 172.21.15.94:0/3009876967 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:38.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f3fff640 1 -- 172.21.15.94:0/3009876967 >> v1:172.21.15.94:6791/0 conn(0x7fe4fc136cc0 legacy=0x7fe4fc1cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe501359640 1 -- 172.21.15.94:0/3009876967 >> v1:172.21.15.94:6792/0 conn(0x7fe4fc1370a0 legacy=0x7fe4fc1d32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe500b58640 1 -- 172.21.15.94:0/3009876967 >> v1:172.21.15.94:6790/0 conn(0x7fe4fc1368e0 legacy=0x7fe4fc1359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949116161 0 0) 0x7fe4fc0a8a70 con 0x7fe4fc1368e0 2024-09-16T10:29:38.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): my global_id is 4331 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e8003520 con 0x7fe4fc1368e0 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636343148 0 0) 0x7fe4fc0a2a70 con 0x7fe4fc136cc0 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): my global_id is 4396 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc0a8a70 con 0x7fe4fc136cc0 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729362980 0 0) 0x7fe4fc0a8620 con 0x7fe4fc1370a0 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 10 monclient(hunting): my global_id is 4344 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.448+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc0a2a70 con 0x7fe4fc1370a0 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2562971430 0 0) 0x7fe4fc0a8a70 con 0x7fe4fc136cc0 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4fc0a8620 con 0x7fe4fc136cc0 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4ec001580 con 0x7fe4fc136cc0 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): dump: 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1489249951 0 0) 0x7fe4fc0a2a70 con 0x7fe4fc1370a0 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4fc0a8a70 con 0x7fe4fc1370a0 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2256841063 0 0) 0x7fe4e8003520 con 0x7fe4fc1368e0 2024-09-16T10:29:38.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4fc0a2a70 con 0x7fe4fc1368e0 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4f4005010 con 0x7fe4fc1370a0 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): dump: 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4f8001d00 con 0x7fe4fc1368e0 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient(hunting): dump: 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.465 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 78755641 0 0) 0x7fe4fc0a8620 con 0x7fe4fc136cc0 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 >> v1:172.21.15.94:6792/0 conn(0x7fe4fc1370a0 legacy=0x7fe4fc1d32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 >> v1:172.21.15.94:6790/0 conn(0x7fe4fc1368e0 legacy=0x7fe4fc1359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 monclient: found mon.c 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc122b60 con 0x7fe4fc136cc0 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe502dbc640 5 monclient: authenticate success, global_id 4396 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe502dbc640 10 monclient: _renew_subs 2024-09-16T10:29:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe502dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc123370 con 0x7fe4fc136cc0 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe502dbc640 10 monclient: _renew_subs 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe502dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc123090 con 0x7fe4fc136cc0 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4ec004dd0 con 0x7fe4fc136cc0 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe4ec005280 con 0x7fe4fc136cc0 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f17fa640 10 monclient: dump: 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.449+0000 7fe4f2ffd640 4 set_mon_vals no callback set 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.450+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fe4ec0102e0 con 0x7fe4fc136cc0 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.450+0000 7fe502dbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.450+0000 7fe502dbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.450+0000 7fe502dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.450+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc0008d0 con 0x7fe4fc136cc0 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.450+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7fe4ec045f30 con 0x7fe4fc136cc0 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.450+0000 7fe500b58640 10 monclient: get_auth_request con 0x7fe4e8039e80 auth_method 0 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.454+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fe4ec0153b0 con 0x7fe4fc136cc0 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.454+0000 7fe4f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.454+0000 7fe4f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.614+0000 7fe502dbc640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T10:29:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.614+0000 7fe502dbc640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T10:29:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.614+0000 7fe502dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.614+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe4fc0a0cd0 con 0x7fe4fc136cc0 2024-09-16T10:29:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.614+0000 7fe4f17fa640 1 -- 172.21.15.94:0/3009876967 <== mon.4 v1:172.21.15.94:6791/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (unknown 3870207488 0 2170607528) 0x7fe4ec013820 con 0x7fe4fc136cc0 2024-09-16T10:29:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.614+0000 7fe4f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T10:29:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.614+0000 7fe4f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:38.616 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-09-16T10:29:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.616+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 >> v1:172.21.15.152:6800/2686360781 conn(0x7fe4e8039e80 legacy=0x7fe4e803c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.616+0000 7fe502dbc640 10 monclient: shutdown 2024-09-16T10:29:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.616+0000 7fe502dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.616+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 >> v1:172.21.15.94:6791/0 conn(0x7fe4fc136cc0 legacy=0x7fe4fc1cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.616+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 shutdown_connections 2024-09-16T10:29:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.616+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 >> 172.21.15.94:0/3009876967 conn(0x7fe4fc0a58d0 msgr2=0x7fe4fc1be2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:38.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.616+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 shutdown_connections 2024-09-16T10:29:38.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.616+0000 7fe502dbc640 1 -- 172.21.15.94:0/3009876967 wait complete. 2024-09-16T10:29:38.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T10:29:38.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.725+0000 7fea79ed0640 10 monclient: get_monmap_and_config 2024-09-16T10:29:38.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.725+0000 7fea79ed0640 10 monclient: build_initial_monmap 2024-09-16T10:29:38.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.725+0000 7fea79ed0640 10 monclient: monmap: 2024-09-16T10:29:38.727 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:38.727 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.727 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:38.726632+0000 2024-09-16T10:29:38.727 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:38.726632+0000 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:38.728 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.726+0000 7fea79ed0640 1 Processor -- start 2024-09-16T10:29:38.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.726+0000 7fea79ed0640 1 -- start start 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.726+0000 7fea79ed0640 10 monclient: init 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 10 monclient: _add_conns ranks=[5,3,4,8,7,2,1,6,0] 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 10 monclient(hunting): picked mon.noname-h con 0x7fea740a3860 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 10 monclient(hunting): picked mon.noname-g con 0x7fea7406c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 10 monclient(hunting): picked mon.noname-c con 0x7fea740a4350 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea74061ae0 con 0x7fea7406c1d0 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740a0600 con 0x7fea740a4350 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740a0b30 con 0x7fea740a3860 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea79ed0640 10 monclient(hunting): authenticate will time out at 746.067165s 2024-09-16T10:29:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea73fff640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fea740a4350 0x7fea7413ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43550/0 (socket says 172.21.15.94:43550) 2024-09-16T10:29:38.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.727+0000 7fea73fff640 1 -- 172.21.15.94:0/2265356683 learned_addr learned my addr 172.21.15.94:0/2265356683 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea73fff640 1 -- 172.21.15.94:0/2265356683 >> v1:172.21.15.94:6791/0 conn(0x7fea740a4350 legacy=0x7fea7413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 1 -- 172.21.15.94:0/2265356683 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356006532 0 0) 0x7fea740a0600 con 0x7fea740a4350 2024-09-16T10:29:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient(hunting): my global_id is 4405 2024-09-16T10:29:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 1 -- 172.21.15.94:0/2265356683 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea5c003520 con 0x7fea740a4350 2024-09-16T10:29:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea737fe640 1 -- 172.21.15.94:0/2265356683 >> v1:172.21.15.152:6791/0 conn(0x7fea740a3860 legacy=0x7fea740a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea72ffd640 1 -- 172.21.15.94:0/2265356683 >> v1:172.21.15.152:6790/0 conn(0x7fea7406c1d0 legacy=0x7fea7413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 1 -- 172.21.15.94:0/2265356683 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 377748031 0 0) 0x7fea5c003520 con 0x7fea740a4350 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 1 -- 172.21.15.94:0/2265356683 >> v1:172.21.15.152:6791/0 conn(0x7fea740a3860 legacy=0x7fea740a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 1 -- 172.21.15.94:0/2265356683 >> v1:172.21.15.152:6790/0 conn(0x7fea7406c1d0 legacy=0x7fea7413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 1 monclient: found mon.noname-c 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 1 -- 172.21.15.94:0/2265356683 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740a0e10 con 0x7fea740a4350 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea79ed0640 5 monclient: authenticate success, global_id 4405 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea79ed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 1 -- 172.21.15.94:0/2265356683 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fea6c001de0 con 0x7fea740a4350 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea727fc640 10 monclient: dump: 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.729+0000 7fea79ed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea727fc640 1 -- 172.21.15.94:0/2265356683 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea6c0020f0 con 0x7fea740a4350 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea727fc640 1 -- 172.21.15.94:0/2265356683 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fea6c004f60 con 0x7fea740a4350 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea79ed0640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea79ed0640 4 set_mon_vals no callback set 2024-09-16T10:29:38.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea727fc640 10 monclient: dump: 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.738 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea79ed0640 10 monclient: shutdown 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea79ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea79ed0640 1 -- 172.21.15.94:0/2265356683 >> v1:172.21.15.94:6791/0 conn(0x7fea740a4350 legacy=0x7fea7413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea79ed0640 1 -- 172.21.15.94:0/2265356683 shutdown_connections 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea79ed0640 1 -- 172.21.15.94:0/2265356683 >> 172.21.15.94:0/2265356683 conn(0x7fea740a5960 msgr2=0x7fea740a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea79ed0640 1 -- 172.21.15.94:0/2265356683 shutdown_connections 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.730+0000 7fea79ed0640 1 -- 172.21.15.94:0/2265356683 wait complete. 2024-09-16T10:29:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.731+0000 7fea79ed0640 10 monclient: build_initial_monmap 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.731+0000 7fea79ed0640 10 monclient: monmap: 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.731+0000 7fea79ed0640 1 Processor -- start 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.731+0000 7fea79ed0640 1 -- start start 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.731+0000 7fea79ed0640 10 monclient: init 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 10 monclient: _add_conns ranks=[3,8,5,0,7,4,2,1,6] 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 10 monclient(hunting): picked mon.noname-g con 0x7fea7406c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 10 monclient(hunting): picked mon.noname-e con 0x7fea740a3860 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 10 monclient(hunting): picked mon.noname-h con 0x7fea740a4350 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7409f4b0 con 0x7fea7406c1d0 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7406b8a0 con 0x7fea740a4350 2024-09-16T10:29:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7409ef60 con 0x7fea740a3860 2024-09-16T10:29:38.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:38.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea79ed0640 10 monclient(hunting): authenticate will time out at 746.072111s 2024-09-16T10:29:38.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea72ffd640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7fea740a3860 0x7fea741cf900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33588/0 (socket says 172.21.15.94:33588) 2024-09-16T10:29:38.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.732+0000 7fea72ffd640 1 -- 172.21.15.94:0/2315139390 learned_addr learned my addr 172.21.15.94:0/2315139390 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:38.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea72ffd640 1 -- 172.21.15.94:0/2315139390 >> v1:172.21.15.94:6793/0 conn(0x7fea740a3860 legacy=0x7fea741cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea737fe640 1 -- 172.21.15.94:0/2315139390 >> v1:172.21.15.152:6790/0 conn(0x7fea7406c1d0 legacy=0x7fea741be380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea73fff640 1 -- 172.21.15.94:0/2315139390 >> v1:172.21.15.152:6791/0 conn(0x7fea740a4350 legacy=0x7fea741d3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:38.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583150864 0 0) 0x7fea7409ef60 con 0x7fea740a3860 2024-09-16T10:29:38.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): my global_id is 4346 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea60003520 con 0x7fea740a3860 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700072724 0 0) 0x7fea7409f4b0 con 0x7fea7406c1d0 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): my global_id is 4449 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7409ef60 con 0x7fea7406c1d0 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629468485 0 0) 0x7fea7406b8a0 con 0x7fea740a4350 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): my global_id is 4406 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7409f4b0 con 0x7fea740a4350 2024-09-16T10:29:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2530772577 0 0) 0x7fea60003520 con 0x7fea740a3860 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea7406b8a0 con 0x7fea740a3860 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fea68003610 con 0x7fea740a3860 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): dump: 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4294685937 0 0) 0x7fea7406b8a0 con 0x7fea740a3860 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 >> v1:172.21.15.152:6791/0 conn(0x7fea740a4350 legacy=0x7fea741d3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 >> v1:172.21.15.152:6790/0 conn(0x7fea7406c1d0 legacy=0x7fea741be380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 monclient: found mon.e 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea741aba60 con 0x7fea740a3860 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient: _finish_auth 0 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea79ed0640 5 monclient: authenticate success, global_id 4346 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea79ed0640 10 monclient: _renew_subs 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea79ed0640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 --> v1:172.21.15.94:6793/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea741abd40 con 0x7fea740a3860 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea79ed0640 10 monclient: _renew_subs 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea79ed0640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea741bea90 con 0x7fea740a3860 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea68003b70 con 0x7fea740a3860 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.734+0000 7fea5bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fea680040a0 con 0x7fea740a3860 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea5bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea5bfff640 10 monclient: dump: 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea71ffb640 4 set_mon_vals no callback set 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea79ed0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea79ed0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea79ed0640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fea68004280 con 0x7fea740a3860 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.735+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 --> v1:172.21.15.94:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea740a2780 con 0x7fea740a3860 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.736+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7fea68046050 con 0x7fea740a3860 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.736+0000 7fea737fe640 10 monclient: get_auth_request con 0x7fea60039140 auth_method 0 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.740+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fea68015530 con 0x7fea740a3860 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.740+0000 7fea5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:38.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.740+0000 7fea5bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:38.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.902+0000 7fea79ed0640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T10:29:38.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.902+0000 7fea79ed0640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T10:29:38.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.902+0000 7fea79ed0640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:38.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.902+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 --> v1:172.21.15.94:6793/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fea741bece0 con 0x7fea740a3860 2024-09-16T10:29:38.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.903+0000 7fea5bfff640 1 -- 172.21.15.94:0/2315139390 <== mon.8 v1:172.21.15.94:6793/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (unknown 116879340 0 265718887) 0x7fea68015120 con 0x7fea740a3860 2024-09-16T10:29:38.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.903+0000 7fea5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T10:29:38.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.903+0000 7fea5bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:38.905 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 8 2024-09-16T10:29:38.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.905+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 >> v1:172.21.15.152:6800/2686360781 conn(0x7fea60039140 legacy=0x7fea6003b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.905+0000 7fea79ed0640 10 monclient: shutdown 2024-09-16T10:29:38.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.905+0000 7fea79ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:38.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.905+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 >> v1:172.21.15.94:6793/0 conn(0x7fea740a3860 legacy=0x7fea741cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:38.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.905+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 shutdown_connections 2024-09-16T10:29:38.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.905+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 >> 172.21.15.94:0/2315139390 conn(0x7fea740a5960 msgr2=0x7fea740a7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:38.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.905+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 shutdown_connections 2024-09-16T10:29:38.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:38.905+0000 7fea79ed0640 1 -- 172.21.15.94:0/2315139390 wait complete. 2024-09-16T10:29:38.916 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T10:29:38.917 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:39.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.013+0000 7f8784da0640 10 monclient: get_monmap_and_config 2024-09-16T10:29:39.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.014+0000 7f8784da0640 10 monclient: build_initial_monmap 2024-09-16T10:29:39.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.014+0000 7f8784da0640 10 monclient: monmap: 2024-09-16T10:29:39.015 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:39.015 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:39.014806+0000 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:39.014806+0000 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:39.016 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:39.017 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.014+0000 7f8784da0640 1 Processor -- start 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.014+0000 7f8784da0640 1 -- start start 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.014+0000 7f8784da0640 10 monclient: init 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 10 monclient: _add_conns ranks=[7,6,3,0,2,8,5,4,1] 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 10 monclient(hunting): picked mon.noname-i con 0x7f87800a28f0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 10 monclient(hunting): picked mon.noname-d con 0x7f87800a2cd0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 10 monclient(hunting): picked mon.noname-g con 0x7f878013c930 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878004f100 con 0x7f878013c930 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878006c580 con 0x7f87800a2cd0 2024-09-16T10:29:39.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8780136600 con 0x7f87800a28f0 2024-09-16T10:29:39.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:39.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f8784da0640 10 monclient(hunting): authenticate will time out at 746.355443s 2024-09-16T10:29:39.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f877effd640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f87800a2cd0 0x7f878013b210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60358/0 (socket says 172.21.15.94:60358) 2024-09-16T10:29:39.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.015+0000 7f877effd640 1 -- 172.21.15.94:0/1046630093 learned_addr learned my addr 172.21.15.94:0/1046630093 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.017+0000 7f877effd640 1 -- 172.21.15.94:0/1046630093 >> v1:172.21.15.94:6792/0 conn(0x7f87800a2cd0 legacy=0x7f878013b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.017+0000 7f877ffff640 1 -- 172.21.15.94:0/1046630093 >> v1:172.21.15.152:6790/0 conn(0x7f878013c930 legacy=0x7f878013ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.017+0000 7f877f7fe640 1 -- 172.21.15.94:0/1046630093 >> v1:172.21.15.152:6792/0 conn(0x7f87800a28f0 legacy=0x7f8780135ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.017+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445086465 0 0) 0x7f878006c580 con 0x7f87800a2cd0 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.017+0000 7f877e7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.017+0000 7f877e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.017+0000 7f877e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.017+0000 7f877e7fc640 10 monclient(hunting): my global_id is 4353 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.017+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8768003520 con 0x7f87800a2cd0 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454923655 0 0) 0x7f878004f100 con 0x7f878013c930 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient(hunting): my global_id is 4458 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f878006c580 con 0x7f878013c930 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408270317 0 0) 0x7f8780136600 con 0x7f87800a28f0 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient(hunting): my global_id is 4372 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f878004f100 con 0x7f87800a28f0 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2880120878 0 0) 0x7f8768003520 con 0x7f87800a2cd0 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 >> v1:172.21.15.152:6792/0 conn(0x7f87800a28f0 legacy=0x7f8780135ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 >> v1:172.21.15.152:6790/0 conn(0x7f878013c930 legacy=0x7f878013ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:39.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 monclient: found mon.noname-d 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878009ab00 con 0x7f87800a2cd0 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8774001a60 con 0x7f87800a2cd0 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 5 monclient: authenticate success, global_id 4353 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: dump: 2024-09-16T10:29:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.025 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8774001cf0 con 0x7f87800a2cd0 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 1 -- 172.21.15.94:0/1046630093 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8774004be0 con 0x7f87800a2cd0 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f877e7fc640 10 monclient: dump: 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.026 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 4 set_mon_vals no callback set 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 10 monclient: shutdown 2024-09-16T10:29:39.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 1 -- 172.21.15.94:0/1046630093 >> v1:172.21.15.94:6792/0 conn(0x7f87800a2cd0 legacy=0x7f878013b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 1 -- 172.21.15.94:0/1046630093 shutdown_connections 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 1 -- 172.21.15.94:0/1046630093 >> 172.21.15.94:0/1046630093 conn(0x7f87800a5730 msgr2=0x7f87800a5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 1 -- 172.21.15.94:0/1046630093 shutdown_connections 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.018+0000 7f8784da0640 1 -- 172.21.15.94:0/1046630093 wait complete. 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.019+0000 7f8784da0640 10 monclient: build_initial_monmap 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.019+0000 7f8784da0640 10 monclient: monmap: 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:39.028 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.019+0000 7f8784da0640 1 Processor -- start 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.019+0000 7f8784da0640 1 -- start start 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.019+0000 7f8784da0640 10 monclient: init 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 10 monclient: _add_conns ranks=[7,3,2,0,8,4,5,1,6] 2024-09-16T10:29:39.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 10 monclient(hunting): picked mon.noname-i con 0x7f87800a28f0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:39.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 10 monclient(hunting): picked mon.noname-g con 0x7f87800a2cd0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:39.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 10 monclient(hunting): picked mon.noname-b con 0x7f878013c930 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:39.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87800a1f00 con 0x7f878013c930 2024-09-16T10:29:39.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878009c560 con 0x7f87800a2cd0 2024-09-16T10:29:39.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878013c3a0 con 0x7f87800a28f0 2024-09-16T10:29:39.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:39.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f8784da0640 10 monclient(hunting): authenticate will time out at 746.360380s 2024-09-16T10:29:39.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f877ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f878013c930 0x7f87801d2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48442/0 (socket says 172.21.15.94:48442) 2024-09-16T10:29:39.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.020+0000 7f877ffff640 1 -- 172.21.15.94:0/1803531267 learned_addr learned my addr 172.21.15.94:0/1803531267 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:39.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f877ffff640 1 -- 172.21.15.94:0/1803531267 >> v1:172.21.15.94:6790/0 conn(0x7f878013c930 legacy=0x7f87801d2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f877f7fe640 1 -- 172.21.15.94:0/1803531267 >> v1:172.21.15.152:6792/0 conn(0x7f87800a28f0 legacy=0x7f878009b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f877effd640 1 -- 172.21.15.94:0/1803531267 >> v1:172.21.15.152:6790/0 conn(0x7f87800a2cd0 legacy=0x7f878009be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769155784 0 0) 0x7f87800a1f00 con 0x7f878013c930 2024-09-16T10:29:39.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): my global_id is 4340 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876c003520 con 0x7f878013c930 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698688272 0 0) 0x7f878013c3a0 con 0x7f87800a28f0 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): my global_id is 4381 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87800a1f00 con 0x7f87800a28f0 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560570307 0 0) 0x7f878009c560 con 0x7f87800a2cd0 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): my global_id is 4467 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f878013c3a0 con 0x7f87800a2cd0 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4184763647 0 0) 0x7f876c003520 con 0x7f878013c930 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f878009c560 con 0x7f878013c930 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f87780020e0 con 0x7f878013c930 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.022+0000 7f8767fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient(hunting): dump: 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.035 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:39.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2086892227 0 0) 0x7f878009c560 con 0x7f878013c930 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 >> v1:172.21.15.152:6792/0 conn(0x7f87800a28f0 legacy=0x7f878009b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 >> v1:172.21.15.152:6790/0 conn(0x7f87800a2cd0 legacy=0x7f878009be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 1 monclient: found mon.b 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878009dc10 con 0x7f878013c930 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient: _finish_auth 0 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8784da0640 5 monclient: authenticate success, global_id 4340 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8784da0640 10 monclient: _renew_subs 2024-09-16T10:29:39.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8784da0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878009ca40 con 0x7f878013c930 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8784da0640 10 monclient: _renew_subs 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8784da0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f878013b680 con 0x7f878013c930 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8778003e10 con 0x7f878013c930 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f87780042c0 con 0x7f878013c930 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f8767fff640 10 monclient: dump: 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.038 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.023+0000 7f877dffb640 4 set_mon_vals no callback set 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.024+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f8778004420 con 0x7f878013c930 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.024+0000 7f8784da0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.024+0000 7f8784da0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.024+0000 7f8784da0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.024+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87800a7f80 con 0x7f878013c930 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.024+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f8778046290 con 0x7f878013c930 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.024+0000 7f877f7fe640 10 monclient: get_auth_request con 0x7f876c0392c0 auth_method 0 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.029+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f87800a7f80 con 0x7f878013c930 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.029+0000 7f8767fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.029+0000 7f8767fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:39.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.190+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f878009b080 con 0x7f876c0392c0 2024-09-16T10:29:39.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.192+0000 7f8767fff640 1 -- 172.21.15.94:0/1803531267 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40978 (unknown 2965378022 0 1951895430) 0x7f8770001a00 con 0x7f876c0392c0 2024-09-16T10:29:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-09-16T10:29:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.194+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 >> v1:172.21.15.152:6800/2686360781 conn(0x7f876c0392c0 legacy=0x7f876c03b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.194+0000 7f8784da0640 10 monclient: shutdown 2024-09-16T10:29:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.194+0000 7f8784da0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.194+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 >> v1:172.21.15.94:6790/0 conn(0x7f878013c930 legacy=0x7f87801d2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.194+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 shutdown_connections 2024-09-16T10:29:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.194+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 >> 172.21.15.94:0/1803531267 conn(0x7f87800a5730 msgr2=0x7f8780134c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.194+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 shutdown_connections 2024-09-16T10:29:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.194+0000 7f8784da0640 1 -- 172.21.15.94:0/1803531267 wait complete. 2024-09-16T10:29:39.206 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-16T10:29:39.006023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176424,"kb_used_data":1864,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054616,"statfs":{"total":579820584960,"available":579639926784,"internally_reserved":0,"allocated":1908736,"data_stored":1316716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.223279"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967479+0000","last_change":"2024-09-16T10:29:32.967697+0000","last_active":"2024-09-16T10:29:32.967479+0000","last_peered":"2024-09-16T10:29:32.967479+0000","last_clean":"2024-09-16T10:29:32.967479+0000","last_became_active":"2024-09-16T10:29:30.052739+0000","last_became_peered":"2024-09-16T10:29:30.052739+0000","last_unstale":"2024-09-16T10:29:32.967479+0000","last_undegraded":"2024-09-16T10:29:32.967479+0000","last_fullsized":"2024-09-16T10:29:32.967479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:38:23.796765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035088200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.789105+0000","last_change":"2024-09-16T10:29:32.789258+0000","last_active":"2024-09-16T10:29:32.789105+0000","last_peered":"2024-09-16T10:29:32.789105+0000","last_clean":"2024-09-16T10:29:32.789105+0000","last_became_active":"2024-09-16T10:29:30.043984+0000","last_became_peered":"2024-09-16T10:29:30.043984+0000","last_unstale":"2024-09-16T10:29:32.789105+0000","last_undegraded":"2024-09-16T10:29:32.789105+0000","last_fullsized":"2024-09-16T10:29:32.789105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:42.075436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028018599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967691+0000","last_change":"2024-09-16T10:29:32.967858+0000","last_active":"2024-09-16T10:29:32.967691+0000","last_peered":"2024-09-16T10:29:32.967691+0000","last_clean":"2024-09-16T10:29:32.967691+0000","last_became_active":"2024-09-16T10:29:30.043883+0000","last_became_peered":"2024-09-16T10:29:30.043883+0000","last_unstale":"2024-09-16T10:29:32.967691+0000","last_undegraded":"2024-09-16T10:29:32.967691+0000","last_fullsized":"2024-09-16T10:29:32.967691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:31.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048663500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.789054+0000","last_change":"2024-09-16T10:29:32.789216+0000","last_active":"2024-09-16T10:29:32.789054+0000","last_peered":"2024-09-16T10:29:32.789054+0000","last_clean":"2024-09-16T10:29:32.789054+0000","last_became_active":"2024-09-16T10:29:30.039372+0000","last_became_peered":"2024-09-16T10:29:30.039372+0000","last_unstale":"2024-09-16T10:29:32.789054+0000","last_undegraded":"2024-09-16T10:29:32.789054+0000","last_fullsized":"2024-09-16T10:29:32.789054+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:15.482545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027726900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.795029+0000","last_change":"2024-09-16T10:29:32.794417+0000","last_active":"2024-09-16T10:29:32.795029+0000","last_peered":"2024-09-16T10:29:32.795029+0000","last_clean":"2024-09-16T10:29:32.795029+0000","last_became_active":"2024-09-16T10:29:30.039308+0000","last_became_peered":"2024-09-16T10:29:30.039308+0000","last_unstale":"2024-09-16T10:29:32.795029+0000","last_undegraded":"2024-09-16T10:29:32.795029+0000","last_fullsized":"2024-09-16T10:29:32.795029+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:04.076079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076855000000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.784986+0000","last_change":"2024-09-16T10:29:32.784986+0000","last_active":"2024-09-16T10:29:32.784986+0000","last_peered":"2024-09-16T10:29:32.784986+0000","last_clean":"2024-09-16T10:29:32.784986+0000","last_became_active":"2024-09-16T10:29:30.039431+0000","last_became_peered":"2024-09-16T10:29:30.039431+0000","last_unstale":"2024-09-16T10:29:32.784986+0000","last_undegraded":"2024-09-16T10:29:32.784986+0000","last_fullsized":"2024-09-16T10:29:32.784986+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:20.175515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041305900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967435+0000","last_change":"2024-09-16T10:29:32.967651+0000","last_active":"2024-09-16T10:29:32.967435+0000","last_peered":"2024-09-16T10:29:32.967435+0000","last_clean":"2024-09-16T10:29:32.967435+0000","last_became_active":"2024-09-16T10:29:30.043748+0000","last_became_peered":"2024-09-16T10:29:30.043748+0000","last_unstale":"2024-09-16T10:29:32.967435+0000","last_undegraded":"2024-09-16T10:29:32.967435+0000","last_fullsized":"2024-09-16T10:29:32.967435+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:37:47.716642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349687,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.794058+0000","last_change":"2024-09-16T10:29:32.794058+0000","last_active":"2024-09-16T10:29:32.794058+0000","last_peered":"2024-09-16T10:29:32.794058+0000","last_clean":"2024-09-16T10:29:32.794058+0000","last_became_active":"2024-09-16T10:29:30.038374+0000","last_became_peered":"2024-09-16T10:29:30.038374+0000","last_unstale":"2024-09-16T10:29:32.794058+0000","last_undegraded":"2024-09-16T10:29:32.794058+0000","last_fullsized":"2024-09-16T10:29:32.794058+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:17.735950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040381400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.420880+0000","last_change":"2024-09-16T10:29:29.005269+0000","last_active":"2024-09-16T10:29:32.420880+0000","last_peered":"2024-09-16T10:29:32.420880+0000","last_clean":"2024-09-16T10:29:32.420880+0000","last_became_active":"2024-09-16T10:29:29.004894+0000","last_became_peered":"2024-09-16T10:29:29.004894+0000","last_unstale":"2024-09-16T10:29:32.420880+0000","last_undegraded":"2024-09-16T10:29:32.420880+0000","last_fullsized":"2024-09-16T10:29:32.420880+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.994140+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:40.518658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":43156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31256,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340584,"statfs":{"total":96636764160,"available":96604758016,"internally_reserved":0,"allocated":139264,"data_stored":43137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]}]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":572065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31264,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340576,"statfs":{"total":96636764160,"available":96604749824,"internally_reserved":0,"allocated":143360,"data_stored":43156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":671744,"data_stored":572065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:39.207 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:39.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.301+0000 7f0cbdc90640 10 monclient: get_monmap_and_config 2024-09-16T10:29:39.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.301+0000 7f0cbdc90640 10 monclient: build_initial_monmap 2024-09-16T10:29:39.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.301+0000 7f0cbdc90640 10 monclient: monmap: 2024-09-16T10:29:39.303 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:39.302746+0000 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:39.302746+0000 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:39.304 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:39.305 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:39.305 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.302+0000 7f0cbdc90640 1 Processor -- start 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.302+0000 7f0cbdc90640 1 -- start start 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.302+0000 7f0cbdc90640 10 monclient: init 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 10 monclient: _add_conns ranks=[3,8,7,0,2,6,1,5,4] 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 10 monclient(hunting): picked mon.noname-g con 0x7f0cb80a0d70 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 10 monclient(hunting): picked mon.noname-e con 0x7f0cb80a1150 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 10 monclient(hunting): picked mon.noname-i con 0x7f0cb813ca20 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb8061ae0 con 0x7f0cb80a0d70 2024-09-16T10:29:39.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb806be10 con 0x7f0cb813ca20 2024-09-16T10:29:39.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb8099570 con 0x7f0cb80a1150 2024-09-16T10:29:39.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:39.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cbdc90640 10 monclient(hunting): authenticate will time out at 746.643227s 2024-09-16T10:29:39.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cb6ffd640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f0cb80a1150 0x7f0cb80a1560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33592/0 (socket says 172.21.15.94:33592) 2024-09-16T10:29:39.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.303+0000 7f0cb6ffd640 1 -- 172.21.15.94:0/650194531 learned_addr learned my addr 172.21.15.94:0/650194531 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:39.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.305+0000 7f0cb6ffd640 1 -- 172.21.15.94:0/650194531 >> v1:172.21.15.94:6793/0 conn(0x7f0cb80a1150 legacy=0x7f0cb80a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.305+0000 7f0cb7fff640 1 -- 172.21.15.94:0/650194531 >> v1:172.21.15.152:6792/0 conn(0x7f0cb813ca20 legacy=0x7f0cb813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.305+0000 7f0cb77fe640 1 -- 172.21.15.94:0/650194531 >> v1:172.21.15.152:6790/0 conn(0x7f0cb80a0d70 legacy=0x7f0cb8099f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.305+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702659362 0 0) 0x7f0cb8099570 con 0x7f0cb80a1150 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.305+0000 7f0cb67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.305+0000 7f0cb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.305+0000 7f0cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.305+0000 7f0cb67fc640 10 monclient(hunting): my global_id is 4355 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.305+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca0003520 con 0x7f0cb80a1150 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183953604 0 0) 0x7f0cb8061ae0 con 0x7f0cb80a0d70 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient(hunting): my global_id is 4476 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb8099570 con 0x7f0cb80a0d70 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939357791 0 0) 0x7f0cb806be10 con 0x7f0cb813ca20 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient(hunting): my global_id is 4390 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb8061ae0 con 0x7f0cb813ca20 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285670436 0 0) 0x7f0ca0003520 con 0x7f0cb80a1150 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 >> v1:172.21.15.152:6792/0 conn(0x7f0cb813ca20 legacy=0x7f0cb813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 >> v1:172.21.15.152:6790/0 conn(0x7f0cb80a0d70 legacy=0x7f0cb8099f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 monclient: found mon.noname-e 2024-09-16T10:29:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb80a2cc0 con 0x7f0cb80a1150 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0cac001a60 con 0x7f0cb80a1150 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 5 monclient: authenticate success, global_id 4355 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: dump: 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.313 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.314 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cac001d70 con 0x7f0cb80a1150 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 1 -- 172.21.15.94:0/650194531 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0cac004be0 con 0x7f0cb80a1150 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 4 set_mon_vals no callback set 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cb67fc640 10 monclient: dump: 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 10 monclient: shutdown 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 1 -- 172.21.15.94:0/650194531 >> v1:172.21.15.94:6793/0 conn(0x7f0cb80a1150 legacy=0x7f0cb80a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 1 -- 172.21.15.94:0/650194531 shutdown_connections 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 1 -- 172.21.15.94:0/650194531 >> 172.21.15.94:0/650194531 conn(0x7f0cb80a1e60 msgr2=0x7f0cb80a2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 1 -- 172.21.15.94:0/650194531 shutdown_connections 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.306+0000 7f0cbdc90640 1 -- 172.21.15.94:0/650194531 wait complete. 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.307+0000 7f0cbdc90640 10 monclient: build_initial_monmap 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.307+0000 7f0cbdc90640 10 monclient: monmap: 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:39.317 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.307+0000 7f0cbdc90640 1 Processor -- start 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.307+0000 7f0cbdc90640 1 -- start start 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.307+0000 7f0cbdc90640 10 monclient: init 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 10 monclient: _add_conns ranks=[5,3,6,4,8,1,7,2,0] 2024-09-16T10:29:39.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 10 monclient(hunting): picked mon.noname-h con 0x7f0cb80a0d70 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:39.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 10 monclient(hunting): picked mon.noname-g con 0x7f0cb80a1150 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:39.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 10 monclient(hunting): picked mon.noname-d con 0x7f0cb813ca20 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:39.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb812b210 con 0x7f0cb80a1150 2024-09-16T10:29:39.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb813fec0 con 0x7f0cb80a0d70 2024-09-16T10:29:39.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb812b740 con 0x7f0cb813ca20 2024-09-16T10:29:39.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:39.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cbdc90640 10 monclient(hunting): authenticate will time out at 746.648318s 2024-09-16T10:29:39.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cb7fff640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f0cb813ca20 0x7f0cb8139f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60368/0 (socket says 172.21.15.94:60368) 2024-09-16T10:29:39.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.308+0000 7f0cb7fff640 1 -- 172.21.15.94:0/1662248928 learned_addr learned my addr 172.21.15.94:0/1662248928 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0cb7fff640 1 -- 172.21.15.94:0/1662248928 >> v1:172.21.15.94:6792/0 conn(0x7f0cb813ca20 legacy=0x7f0cb8139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657774812 0 0) 0x7f0cb812b740 con 0x7f0cb813ca20 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0cb6ffd640 1 -- 172.21.15.94:0/1662248928 >> v1:172.21.15.152:6790/0 conn(0x7f0cb80a1150 legacy=0x7f0cb8139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): my global_id is 4362 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0cb77fe640 1 -- 172.21.15.94:0/1662248928 >> v1:172.21.15.152:6791/0 conn(0x7f0cb80a0d70 legacy=0x7f0cb8139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca4003520 con 0x7f0cb813ca20 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4211332563 0 0) 0x7f0ca4003520 con 0x7f0cb813ca20 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cb812b740 con 0x7f0cb813ca20 2024-09-16T10:29:39.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0cb0002030 con 0x7f0cb813ca20 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): dump: 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.323 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73648404 0 0) 0x7f0cb813fec0 con 0x7f0cb80a0d70 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.310+0000 7f0c9ffff640 10 monclient(hunting): my global_id is 4415 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca4003520 con 0x7f0cb80a0d70 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047433541 0 0) 0x7f0cb812b210 con 0x7f0cb80a1150 2024-09-16T10:29:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient(hunting): my global_id is 4485 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb813fec0 con 0x7f0cb80a1150 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3432283772 0 0) 0x7f0cb812b740 con 0x7f0cb813ca20 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 >> v1:172.21.15.152:6791/0 conn(0x7f0cb80a0d70 legacy=0x7f0cb8139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 >> v1:172.21.15.152:6790/0 conn(0x7f0cb80a1150 legacy=0x7f0cb8139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 monclient: found mon.d 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb812ba20 con 0x7f0cb813ca20 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient: _finish_auth 0 2024-09-16T10:29:39.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0cbdc90640 5 monclient: authenticate success, global_id 4362 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0cbdc90640 10 monclient: _renew_subs 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0cbdc90640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb813ac50 con 0x7f0cb813ca20 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0cbdc90640 10 monclient: _renew_subs 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0cbdc90640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb809adc0 con 0x7f0cb813ca20 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cb0003e00 con 0x7f0cb813ca20 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0cb00039d0 con 0x7f0cb813ca20 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:39.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 10 monclient: dump: 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.327 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0cb5ffb640 4 set_mon_vals no callback set 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.311+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f0cb0004370 con 0x7f0cb813ca20 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.312+0000 7f0cbdc90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.312+0000 7f0cbdc90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.312+0000 7f0cbdc90640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.312+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb812a4c0 con 0x7f0cb813ca20 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.312+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f0cb00462f0 con 0x7f0cb813ca20 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.312+0000 7f0cb77fe640 10 monclient: get_auth_request con 0x7f0ca4039230 auth_method 0 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.316+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f0cb0015540 con 0x7f0cb813ca20 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.316+0000 7f0c9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.316+0000 7f0c9ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:39.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.476+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cb80a4700 con 0x7f0ca4039230 2024-09-16T10:29:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.478+0000 7f0c9ffff640 1 -- 172.21.15.94:0/1662248928 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40978 (unknown 2965378022 0 1951895430) 0x7f0cb80a4700 con 0x7f0ca4039230 2024-09-16T10:29:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:29:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-09-16T10:29:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.480+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 >> v1:172.21.15.152:6800/2686360781 conn(0x7f0ca4039230 legacy=0x7f0ca403b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.480+0000 7f0cbdc90640 10 monclient: shutdown 2024-09-16T10:29:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.480+0000 7f0cbdc90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.480+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 >> v1:172.21.15.94:6792/0 conn(0x7f0cb813ca20 legacy=0x7f0cb8139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.480+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 shutdown_connections 2024-09-16T10:29:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.480+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 >> 172.21.15.94:0/1662248928 conn(0x7f0cb80a1e60 msgr2=0x7f0cb813c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.480+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 shutdown_connections 2024-09-16T10:29:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.480+0000 7f0cbdc90640 1 -- 172.21.15.94:0/1662248928 wait complete. 2024-09-16T10:29:39.492 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-16T10:29:39.006023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176424,"kb_used_data":1864,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054616,"statfs":{"total":579820584960,"available":579639926784,"internally_reserved":0,"allocated":1908736,"data_stored":1316716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.223279"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967479+0000","last_change":"2024-09-16T10:29:32.967697+0000","last_active":"2024-09-16T10:29:32.967479+0000","last_peered":"2024-09-16T10:29:32.967479+0000","last_clean":"2024-09-16T10:29:32.967479+0000","last_became_active":"2024-09-16T10:29:30.052739+0000","last_became_peered":"2024-09-16T10:29:30.052739+0000","last_unstale":"2024-09-16T10:29:32.967479+0000","last_undegraded":"2024-09-16T10:29:32.967479+0000","last_fullsized":"2024-09-16T10:29:32.967479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:38:23.796765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035088200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.789105+0000","last_change":"2024-09-16T10:29:32.789258+0000","last_active":"2024-09-16T10:29:32.789105+0000","last_peered":"2024-09-16T10:29:32.789105+0000","last_clean":"2024-09-16T10:29:32.789105+0000","last_became_active":"2024-09-16T10:29:30.043984+0000","last_became_peered":"2024-09-16T10:29:30.043984+0000","last_unstale":"2024-09-16T10:29:32.789105+0000","last_undegraded":"2024-09-16T10:29:32.789105+0000","last_fullsized":"2024-09-16T10:29:32.789105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:42.075436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028018599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967691+0000","last_change":"2024-09-16T10:29:32.967858+0000","last_active":"2024-09-16T10:29:32.967691+0000","last_peered":"2024-09-16T10:29:32.967691+0000","last_clean":"2024-09-16T10:29:32.967691+0000","last_became_active":"2024-09-16T10:29:30.043883+0000","last_became_peered":"2024-09-16T10:29:30.043883+0000","last_unstale":"2024-09-16T10:29:32.967691+0000","last_undegraded":"2024-09-16T10:29:32.967691+0000","last_fullsized":"2024-09-16T10:29:32.967691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:31.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048663500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.789054+0000","last_change":"2024-09-16T10:29:32.789216+0000","last_active":"2024-09-16T10:29:32.789054+0000","last_peered":"2024-09-16T10:29:32.789054+0000","last_clean":"2024-09-16T10:29:32.789054+0000","last_became_active":"2024-09-16T10:29:30.039372+0000","last_became_peered":"2024-09-16T10:29:30.039372+0000","last_unstale":"2024-09-16T10:29:32.789054+0000","last_undegraded":"2024-09-16T10:29:32.789054+0000","last_fullsized":"2024-09-16T10:29:32.789054+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:15.482545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027726900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.795029+0000","last_change":"2024-09-16T10:29:32.794417+0000","last_active":"2024-09-16T10:29:32.795029+0000","last_peered":"2024-09-16T10:29:32.795029+0000","last_clean":"2024-09-16T10:29:32.795029+0000","last_became_active":"2024-09-16T10:29:30.039308+0000","last_became_peered":"2024-09-16T10:29:30.039308+0000","last_unstale":"2024-09-16T10:29:32.795029+0000","last_undegraded":"2024-09-16T10:29:32.795029+0000","last_fullsized":"2024-09-16T10:29:32.795029+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:04.076079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076855000000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.784986+0000","last_change":"2024-09-16T10:29:32.784986+0000","last_active":"2024-09-16T10:29:32.784986+0000","last_peered":"2024-09-16T10:29:32.784986+0000","last_clean":"2024-09-16T10:29:32.784986+0000","last_became_active":"2024-09-16T10:29:30.039431+0000","last_became_peered":"2024-09-16T10:29:30.039431+0000","last_unstale":"2024-09-16T10:29:32.784986+0000","last_undegraded":"2024-09-16T10:29:32.784986+0000","last_fullsized":"2024-09-16T10:29:32.784986+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:20.175515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041305900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.967435+0000","last_change":"2024-09-16T10:29:32.967651+0000","last_active":"2024-09-16T10:29:32.967435+0000","last_peered":"2024-09-16T10:29:32.967435+0000","last_clean":"2024-09-16T10:29:32.967435+0000","last_became_active":"2024-09-16T10:29:30.043748+0000","last_became_peered":"2024-09-16T10:29:30.043748+0000","last_unstale":"2024-09-16T10:29:32.967435+0000","last_undegraded":"2024-09-16T10:29:32.967435+0000","last_fullsized":"2024-09-16T10:29:32.967435+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:37:47.716642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349687,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.794058+0000","last_change":"2024-09-16T10:29:32.794058+0000","last_active":"2024-09-16T10:29:32.794058+0000","last_peered":"2024-09-16T10:29:32.794058+0000","last_clean":"2024-09-16T10:29:32.794058+0000","last_became_active":"2024-09-16T10:29:30.038374+0000","last_became_peered":"2024-09-16T10:29:30.038374+0000","last_unstale":"2024-09-16T10:29:32.794058+0000","last_undegraded":"2024-09-16T10:29:32.794058+0000","last_fullsized":"2024-09-16T10:29:32.794058+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:17.735950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040381400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T10:29:32.420880+0000","last_change":"2024-09-16T10:29:29.005269+0000","last_active":"2024-09-16T10:29:32.420880+0000","last_peered":"2024-09-16T10:29:32.420880+0000","last_clean":"2024-09-16T10:29:32.420880+0000","last_became_active":"2024-09-16T10:29:29.004894+0000","last_became_peered":"2024-09-16T10:29:29.004894+0000","last_unstale":"2024-09-16T10:29:32.420880+0000","last_undegraded":"2024-09-16T10:29:32.420880+0000","last_fullsized":"2024-09-16T10:29:32.420880+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.994140+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:40.518658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":43156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31256,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340584,"statfs":{"total":96636764160,"available":96604758016,"internally_reserved":0,"allocated":139264,"data_stored":43137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]}]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":572065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31264,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340576,"statfs":{"total":96636764160,"available":96604749824,"internally_reserved":0,"allocated":143360,"data_stored":43156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":671744,"data_stored":572065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:39.493 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T10:29:39.494 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T10:29:39.494 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T10:29:39.503 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T10:29:39.503 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:29:39.503 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20' 2024-09-16T10:29:39.504 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T10:29:39.504 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726482579, revive delay: 90.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-16T10:29:39.504 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-16T10:29:39.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:29:39.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.599+0000 7f485b073640 10 monclient: get_monmap_and_config 2024-09-16T10:29:39.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.599+0000 7f485b073640 10 monclient: build_initial_monmap 2024-09-16T10:29:39.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.599+0000 7f485b073640 10 monclient: monmap: 2024-09-16T10:29:39.601 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:39.601 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.601 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:39.600465+0000 2024-09-16T10:29:39.601 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:39.600465+0000 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:39.602 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 1 Processor -- start 2024-09-16T10:29:39.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 1 -- start start 2024-09-16T10:29:39.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 10 monclient: init 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 10 monclient: _add_conns ranks=[8,0,3,7,1,6,5,2,4] 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 10 monclient(hunting): picked mon.noname-e con 0x7f48540a3860 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 10 monclient(hunting): picked mon.noname-a con 0x7f485406c1d0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 10 monclient(hunting): picked mon.noname-g con 0x7f48540a4350 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4854061ae0 con 0x7f485406c1d0 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48540a0600 con 0x7f48540a4350 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48540a0b30 con 0x7f48540a3860 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.600+0000 7f485b073640 10 monclient(hunting): authenticate will time out at 746.940959s 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.601+0000 7f4858e0f640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f48540a3860 0x7f48540a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33608/0 (socket says 172.21.15.94:33608) 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.601+0000 7f4858e0f640 1 -- 172.21.15.94:0/3169448438 learned_addr learned my addr 172.21.15.94:0/3169448438 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:39.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f4858e0f640 1 -- 172.21.15.94:0/3169448438 >> v1:172.21.15.94:6793/0 conn(0x7f48540a3860 legacy=0x7f48540a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484bfff640 1 -- 172.21.15.94:0/3169448438 >> v1:172.21.15.94:6789/0 conn(0x7f485406c1d0 legacy=0x7f485413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 -- 172.21.15.94:0/3169448438 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395047206 0 0) 0x7f48540a0b30 con 0x7f48540a3860 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient(hunting): my global_id is 4373 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 -- 172.21.15.94:0/3169448438 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f483c003520 con 0x7f48540a3860 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f4859610640 1 -- 172.21.15.94:0/3169448438 >> v1:172.21.15.152:6790/0 conn(0x7f48540a4350 legacy=0x7f485413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 -- 172.21.15.94:0/3169448438 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4190938963 0 0) 0x7f483c003520 con 0x7f48540a3860 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 -- 172.21.15.94:0/3169448438 >> v1:172.21.15.152:6790/0 conn(0x7f48540a4350 legacy=0x7f485413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 -- 172.21.15.94:0/3169448438 >> v1:172.21.15.94:6789/0 conn(0x7f485406c1d0 legacy=0x7f485413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 monclient: found mon.noname-e 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 -- 172.21.15.94:0/3169448438 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48540a0e10 con 0x7f48540a3860 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 -- 172.21.15.94:0/3169448438 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4850001a60 con 0x7f48540a3860 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f485b073640 5 monclient: authenticate success, global_id 4373 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f485b073640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: dump: 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.610 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f485b073640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 -- 172.21.15.94:0/3169448438 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4850001d70 con 0x7f48540a3860 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 1 -- 172.21.15.94:0/3169448438 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4850004be0 con 0x7f48540a3860 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f485b073640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f485b073640 4 set_mon_vals no callback set 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.603+0000 7f484b7fe640 10 monclient: dump: 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.612 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.604+0000 7f485b073640 10 monclient: shutdown 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.604+0000 7f485b073640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.604+0000 7f485b073640 1 -- 172.21.15.94:0/3169448438 >> v1:172.21.15.94:6793/0 conn(0x7f48540a3860 legacy=0x7f48540a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.604+0000 7f485b073640 1 -- 172.21.15.94:0/3169448438 shutdown_connections 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.604+0000 7f485b073640 1 -- 172.21.15.94:0/3169448438 >> 172.21.15.94:0/3169448438 conn(0x7f48540a5960 msgr2=0x7f48540a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.604+0000 7f485b073640 1 -- 172.21.15.94:0/3169448438 shutdown_connections 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.604+0000 7f485b073640 1 -- 172.21.15.94:0/3169448438 wait complete. 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.604+0000 7f485b073640 10 monclient: build_initial_monmap 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.604+0000 7f485b073640 10 monclient: monmap: 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:39.613 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:39.614 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 1 Processor -- start 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 1 -- start start 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 10 monclient: init 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 10 monclient: _add_conns ranks=[5,7,8,3,2,4,1,0,6] 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 10 monclient(hunting): picked mon.noname-h con 0x7f485406c1d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 10 monclient(hunting): picked mon.noname-i con 0x7f48540a3860 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 10 monclient(hunting): picked mon.noname-e con 0x7f48540a4350 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485409f320 con 0x7f485406c1d0 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485406bae0 con 0x7f48540a3860 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48541cb8b0 con 0x7f48540a4350 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f485b073640 10 monclient(hunting): authenticate will time out at 746.945776s 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f4859610640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f48540a4350 0x7f48541d3400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33614/0 (socket says 172.21.15.94:33614) 2024-09-16T10:29:39.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.605+0000 7f4859610640 1 -- 172.21.15.94:0/1855160132 learned_addr learned my addr 172.21.15.94:0/1855160132 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.607+0000 7f4859610640 1 -- 172.21.15.94:0/1855160132 >> v1:172.21.15.94:6793/0 conn(0x7f48540a4350 legacy=0x7f48541d3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274727481 0 0) 0x7f48541cb8b0 con 0x7f48540a4350 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): my global_id is 4382 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840003520 con 0x7f48540a4350 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f4858e0f640 1 -- 172.21.15.94:0/1855160132 >> v1:172.21.15.152:6791/0 conn(0x7f485406c1d0 legacy=0x7f48541be740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f484bfff640 1 -- 172.21.15.94:0/1855160132 >> v1:172.21.15.152:6792/0 conn(0x7f48540a3860 legacy=0x7f48541cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f4859610640 1 --1- 172.21.15.94:0/1855160132 >> v1:172.21.15.94:6793/0 conn(0x7f48540a4350 0x7f48541d3400 :-1 s=OPENED pgs=69 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7f484c002050 mon_map magic: 0 v1 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1150398356 0 0) 0x7f4840003520 con 0x7f48540a4350 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48541cb8b0 con 0x7f48540a4350 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212398221 0 0) 0x7f485406bae0 con 0x7f48540a3860 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): my global_id is 4408 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840003520 con 0x7f48540a3860 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933634143 0 0) 0x7f485409f320 con 0x7f485406c1d0 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): my global_id is 4442 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485406bae0 con 0x7f485406c1d0 2024-09-16T10:29:39.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f484c002050 con 0x7f48540a4350 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): dump: 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.621 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3187576960 0 0) 0x7f4840003520 con 0x7f48540a3860 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f485409f320 con 0x7f48540a3860 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4844001c20 con 0x7f48540a3860 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): dump: 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.622 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141107478 0 0) 0x7f485406bae0 con 0x7f485406c1d0 2024-09-16T10:29:39.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4840003520 con 0x7f485406c1d0 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4850004b20 con 0x7f485406c1d0 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient(hunting): dump: 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.624 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1182313475 0 0) 0x7f485409f320 con 0x7f48540a3860 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 >> v1:172.21.15.94:6793/0 conn(0x7f48540a4350 legacy=0x7f48541d3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 >> v1:172.21.15.152:6791/0 conn(0x7f485406c1d0 legacy=0x7f48541be740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 monclient: found mon.i 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:39.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48541ac100 con 0x7f48540a3860 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1729356389 0 0) 0x7f48541cb8b0 con 0x7f48540a4350 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f485b073640 5 monclient: authenticate success, global_id 4408 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f485b073640 10 monclient: _renew_subs 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f485b073640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485409f870 con 0x7f48540a3860 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f485b073640 10 monclient: _renew_subs 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f485b073640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48541aa6c0 con 0x7f48540a3860 2024-09-16T10:29:39.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48440038c0 con 0x7f48540a3860 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4844003df0 con 0x7f48540a3860 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f484affd640 4 set_mon_vals no callback set 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.608+0000 7f48497fa640 10 monclient: dump: 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.609+0000 7f485b073640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.609+0000 7f485b073640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.609+0000 7f485b073640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.609+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f4844003fd0 con 0x7f48540a3860 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.609+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4834005040 con 0x7f48540a3860 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.609+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f4844045df0 con 0x7f48540a3860 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.610+0000 7f4858e0f640 10 monclient: get_auth_request con 0x7f4840039cb0 auth_method 0 2024-09-16T10:29:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.614+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f48440152d0 con 0x7f48540a3860 2024-09-16T10:29:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.614+0000 7f48497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.614+0000 7f48497fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:39.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.839+0000 7f485b073640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:29:39.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.839+0000 7f485b073640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:29:39.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.839+0000 7f485b073640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:39.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.839+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4834005870 con 0x7f48540a3860 2024-09-16T10:29:39.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.840+0000 7f48497fa640 1 -- 172.21.15.94:0/1855160132 <== mon.7 v1:172.21.15.152:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+2712 (unknown 254746657 0 2123981568) 0x7f484401d8c0 con 0x7f48540a3860 2024-09-16T10:29:39.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.840+0000 7f48497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:29:39.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.840+0000 7f48497fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:39.842 INFO:teuthology.orchestra.run.smithi094.stdout:{"election_epoch":8,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","modified":"2024-09-16T10:28:46.257482Z","created":"2024-09-16T10:28:46.257482Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6790","nonce":0}]},"addr":"172.21.15.94:6790/0","public_addr":"172.21.15.94:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6790","nonce":0}]},"addr":"172.21.15.152:6790/0","public_addr":"172.21.15.152:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6791","nonce":0}]},"addr":"172.21.15.94:6791/0","public_addr":"172.21.15.94:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6791","nonce":0}]},"addr":"172.21.15.152:6791/0","public_addr":"172.21.15.152:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6792","nonce":0}]},"addr":"172.21.15.94:6792/0","public_addr":"172.21.15.94:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6792","nonce":0}]},"addr":"172.21.15.152:6792/0","public_addr":"172.21.15.152:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6793","nonce":0}]},"addr":"172.21.15.94:6793/0","public_addr":"172.21.15.94:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:29:39.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.842+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 >> v1:172.21.15.152:6800/2686360781 conn(0x7f4840039cb0 legacy=0x7f484003c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.842+0000 7f485b073640 10 monclient: shutdown 2024-09-16T10:29:39.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.842+0000 7f485b073640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:39.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.842+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 >> v1:172.21.15.152:6792/0 conn(0x7f48540a3860 legacy=0x7f48541cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.842+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 shutdown_connections 2024-09-16T10:29:39.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.842+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 >> 172.21.15.94:0/1855160132 conn(0x7f48540a5960 msgr2=0x7f48540a7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:39.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.842+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 shutdown_connections 2024-09-16T10:29:39.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.842+0000 7f485b073640 1 -- 172.21.15.94:0/1855160132 wait complete. 2024-09-16T10:29:39.853 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-16T10:29:39.854 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T10:29:39.854 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-16T10:29:39.854 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:29:39.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.950+0000 7f96c5eb1640 10 monclient: get_monmap_and_config 2024-09-16T10:29:39.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.950+0000 7f96c5eb1640 10 monclient: build_initial_monmap 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.950+0000 7f96c5eb1640 10 monclient: monmap: 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:39.951044+0000 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:39.951044+0000 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:39.953 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.950+0000 7f96c5eb1640 1 Processor -- start 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 1 -- start start 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 10 monclient: init 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 10 monclient: _add_conns ranks=[7,3,8,0,5,4,2,6,1] 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 10 monclient(hunting): picked mon.noname-i con 0x7f96c00a38d0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 10 monclient(hunting): picked mon.noname-g con 0x7f96c006c670 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 10 monclient(hunting): picked mon.noname-e con 0x7f96c00a43c0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c0061ae0 con 0x7f96c006c670 2024-09-16T10:29:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c00a04d0 con 0x7f96c00a38d0 2024-09-16T10:29:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c00a3250 con 0x7f96c00a43c0 2024-09-16T10:29:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96c5eb1640 10 monclient(hunting): authenticate will time out at 747.291559s 2024-09-16T10:29:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96bffff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f96c00a43c0 0x7f96c013f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33620/0 (socket says 172.21.15.94:33620) 2024-09-16T10:29:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.951+0000 7f96bffff640 1 -- 172.21.15.94:0/2395822542 learned_addr learned my addr 172.21.15.94:0/2395822542 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:39.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.953+0000 7f96bffff640 1 -- 172.21.15.94:0/2395822542 >> v1:172.21.15.94:6793/0 conn(0x7f96c00a43c0 legacy=0x7f96c013f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.953+0000 7f96beffd640 1 -- 172.21.15.94:0/2395822542 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938579091 0 0) 0x7f96c00a3250 con 0x7f96c00a43c0 2024-09-16T10:29:39.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient(hunting): my global_id is 4391 2024-09-16T10:29:39.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 1 -- 172.21.15.94:0/2395822542 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac003520 con 0x7f96c00a43c0 2024-09-16T10:29:39.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96b6ffd640 1 -- 172.21.15.94:0/2395822542 >> v1:172.21.15.152:6790/0 conn(0x7f96c006c670 legacy=0x7f96c013bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96bf7fe640 1 -- 172.21.15.94:0/2395822542 >> v1:172.21.15.152:6792/0 conn(0x7f96c00a38d0 legacy=0x7f96c00a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 1 -- 172.21.15.94:0/2395822542 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2232587491 0 0) 0x7f96ac003520 con 0x7f96c00a43c0 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 1 -- 172.21.15.94:0/2395822542 >> v1:172.21.15.152:6792/0 conn(0x7f96c00a38d0 legacy=0x7f96c00a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 1 -- 172.21.15.94:0/2395822542 >> v1:172.21.15.152:6790/0 conn(0x7f96c006c670 legacy=0x7f96c013bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 1 monclient: found mon.noname-e 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 1 -- 172.21.15.94:0/2395822542 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c00a0a00 con 0x7f96c00a43c0 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 1 -- 172.21.15.94:0/2395822542 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f96b8001ea0 con 0x7f96c00a43c0 2024-09-16T10:29:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 5 monclient: authenticate success, global_id 4391 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: dump: 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 1 -- 172.21.15.94:0/2395822542 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96b8004c20 con 0x7f96c00a43c0 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 1 -- 172.21.15.94:0/2395822542 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f96b80050d0 con 0x7f96c00a43c0 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 4 set_mon_vals no callback set 2024-09-16T10:29:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96beffd640 10 monclient: dump: 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 10 monclient: shutdown 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 1 -- 172.21.15.94:0/2395822542 >> v1:172.21.15.94:6793/0 conn(0x7f96c00a43c0 legacy=0x7f96c013f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 1 -- 172.21.15.94:0/2395822542 shutdown_connections 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 1 -- 172.21.15.94:0/2395822542 >> 172.21.15.94:0/2395822542 conn(0x7f96c00a59d0 msgr2=0x7f96c00a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 1 -- 172.21.15.94:0/2395822542 shutdown_connections 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.954+0000 7f96c5eb1640 1 -- 172.21.15.94:0/2395822542 wait complete. 2024-09-16T10:29:39.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.955+0000 7f96c5eb1640 10 monclient: build_initial_monmap 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.955+0000 7f96c5eb1640 10 monclient: monmap: 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:39.964 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.955+0000 7f96c5eb1640 1 Processor -- start 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 1 -- start start 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 10 monclient: init 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 10 monclient: _add_conns ranks=[1,2,8,6,4,3,0,7,5] 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 10 monclient(hunting): picked mon.noname-f con 0x7f96c006c670 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 10 monclient(hunting): picked mon.noname-b con 0x7f96c00a38d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 10 monclient(hunting): picked mon.noname-e con 0x7f96c00a43c0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:39.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c009ef80 con 0x7f96c006c670 2024-09-16T10:29:39.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c006ba90 con 0x7f96c00a38d0 2024-09-16T10:29:39.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c009f2b0 con 0x7f96c00a43c0 2024-09-16T10:29:39.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:39.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96c5eb1640 10 monclient(hunting): authenticate will time out at 747.296490s 2024-09-16T10:29:39.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96bffff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f96c00a43c0 0x7f96c01d3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33632/0 (socket says 172.21.15.94:33632) 2024-09-16T10:29:39.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96bffff640 1 -- 172.21.15.94:0/1032502138 learned_addr learned my addr 172.21.15.94:0/1032502138 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:39.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.956+0000 7f96b6ffd640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f96c00a38d0 0x7f96c01cfe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48450/0 (socket says 172.21.15.94:48450) 2024-09-16T10:29:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96b6ffd640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6790/0 conn(0x7f96c00a38d0 legacy=0x7f96c01cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bffff640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6793/0 conn(0x7f96c00a43c0 legacy=0x7f96c01d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434947793 0 0) 0x7f96c009f2b0 con 0x7f96c00a43c0 2024-09-16T10:29:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 10 monclient(hunting): my global_id is 4400 2024-09-16T10:29:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a0003520 con 0x7f96c00a43c0 2024-09-16T10:29:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131695710 0 0) 0x7f96c006ba90 con 0x7f96c00a38d0 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 10 monclient(hunting): my global_id is 4349 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c009f2b0 con 0x7f96c00a38d0 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.958+0000 7f96bf7fe640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.152:6789/0 conn(0x7f96c006c670 legacy=0x7f96c0136820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4274997444 0 0) 0x7f96a0003520 con 0x7f96c00a43c0 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96c006ba90 con 0x7f96c00a43c0 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f96b80050d0 con 0x7f96c00a43c0 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): dump: 2024-09-16T10:29:39.971 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.972 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 394958404 0 0) 0x7f96c009f2b0 con 0x7f96c00a38d0 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96a0003520 con 0x7f96c00a38d0 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f96a8001d10 con 0x7f96c00a38d0 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): dump: 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.973 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647793138 0 0) 0x7f96c009ef80 con 0x7f96c006c670 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient(hunting): my global_id is 4366 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c009f2b0 con 0x7f96c006c670 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2400233340 0 0) 0x7f96c006ba90 con 0x7f96c00a43c0 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6790/0 conn(0x7f96c00a38d0 legacy=0x7f96c01cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.152:6789/0 conn(0x7f96c006c670 legacy=0x7f96c0136820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient: _finish_hunting 0 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 monclient: found mon.e 2024-09-16T10:29:39.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c01241e0 con 0x7f96c00a43c0 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient: _finish_auth 0 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96c5eb1640 5 monclient: authenticate success, global_id 4400 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96c5eb1640 10 monclient: _renew_subs 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96c5eb1640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6793/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c0123940 con 0x7f96c00a43c0 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96c5eb1640 10 monclient: _renew_subs 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96c5eb1640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c0136f30 con 0x7f96c00a43c0 2024-09-16T10:29:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.8 v1:172.21.15.94:6793/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96b8001e80 con 0x7f96c00a43c0 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.8 v1:172.21.15.94:6793/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f96b80058d0 con 0x7f96c00a43c0 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96be7fc640 4 set_mon_vals no callback set 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.959+0000 7f96bcff9640 10 monclient: dump: 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.977 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.960+0000 7f96c5eb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.960+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f96a4006580 con 0x7f96a4002820 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.960+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.8 v1:172.21.15.94:6793/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f96b8010930 con 0x7f96c00a43c0 2024-09-16T10:29:39.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.960+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.8 v1:172.21.15.94:6793/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (unknown 3191044782 0 0) 0x7f96b8046490 con 0x7f96c00a43c0 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.960+0000 7f96b6ffd640 10 monclient: get_auth_request con 0x7f96a0039800 auth_method 0 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bf7fe640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6789/0 conn(0x7f96a4002820 legacy=0x7f96a4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271590693 0 0) 0x7f96a4006580 con 0x7f96a4002820 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 10 monclient(hunting): my global_id is 4419 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c006ba90 con 0x7f96a4002820 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3778891805 0 0) 0x7f96c006ba90 con 0x7f96a4002820 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96a4006580 con 0x7f96a4002820 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f96b0001bb0 con 0x7f96a4002820 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 10 monclient: dump: 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.980 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.981 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.981 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 708393421 0 0) 0x7f96a4006580 con 0x7f96a4002820 2024-09-16T10:29:39.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.962+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96a400a0d0 con 0x7f96a4002820 2024-09-16T10:29:39.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.963+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f96b0006830 con 0x7f96a4002820 2024-09-16T10:29:39.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.963+0000 7f96bcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:39.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.963+0000 7f96bcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:39.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.963+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6789/0 conn(0x7f96a4002820 legacy=0x7f96a4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.963+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6789/0 conn(0x7f96a4002820 legacy=0x7f96a4004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.969+0000 7f96c5eb1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.969+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f96a4009d70 con 0x7f96a4009090 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.971+0000 7f96bf7fe640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6789/0 conn(0x7f96a4009090 legacy=0x7f96a4009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.971+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910204889 0 0) 0x7f96a4009d70 con 0x7f96a4009090 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.971+0000 7f96bcff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.971+0000 7f96bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.971+0000 7f96bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.971+0000 7f96bcff9640 10 monclient(hunting): my global_id is 4428 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.971+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a4006580 con 0x7f96a4009090 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.971+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1818662322 0 0) 0x7f96a4006580 con 0x7f96a4009090 2024-09-16T10:29:39.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96a4009d70 con 0x7f96a4009090 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f96b0003ec0 con 0x7f96a4009090 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 10 monclient: dump: 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:39.984 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2866268876 0 0) 0x7f96a4009d70 con 0x7f96a4009090 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 --> v1:172.21.15.94:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f96a400c280 con 0x7f96a4009090 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 <== mon.0 v1:172.21.15.94:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7454 (unknown 0 0 3878883543) 0x7f96b0004420 con 0x7f96a4009090 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:39.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6789/0 conn(0x7f96a4009090 legacy=0x7f96a4009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bcff9640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6789/0 conn(0x7f96a4009090 legacy=0x7f96a4009520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "leader", 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 8, 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:29:39.986 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 23, 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:39.987 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:39.988 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:29:39.989 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:29:39.990 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:39.991 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:39.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:29:39.995 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:29:39.996 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:39.998 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:39.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.000 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.002 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.003 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.004 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 3 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:29:40.005 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:29:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.152:6800/2686360781 conn(0x7f96a0039800 legacy=0x7f96a003bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96c5eb1640 10 monclient: shutdown 2024-09-16T10:29:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96c5eb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 >> v1:172.21.15.94:6793/0 conn(0x7f96c00a43c0 legacy=0x7f96c01d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bffff640 1 -- 172.21.15.94:0/1032502138 reap_dead start 2024-09-16T10:29:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96bffff640 1 -- 172.21.15.94:0/1032502138 reap_dead start 2024-09-16T10:29:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 shutdown_connections 2024-09-16T10:29:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 >> 172.21.15.94:0/1032502138 conn(0x7f96c00a59d0 msgr2=0x7f96c00a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 shutdown_connections 2024-09-16T10:29:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:39.972+0000 7f96c5eb1640 1 -- 172.21.15.94:0/1032502138 wait complete. 2024-09-16T10:29:40.007 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:29:40.007 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-16T10:29:40.007 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:29:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.110+0000 7f11d53c3640 10 monclient: get_monmap_and_config 2024-09-16T10:29:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.110+0000 7f11d53c3640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.111+0000 7f11d53c3640 10 monclient: monmap: 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:40.111787+0000 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:40.111787+0000 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.114 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.111+0000 7f11d53c3640 1 Processor -- start 2024-09-16T10:29:40.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.111+0000 7f11d53c3640 1 -- start start 2024-09-16T10:29:40.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.111+0000 7f11d53c3640 10 monclient: init 2024-09-16T10:29:40.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 10 monclient: _add_conns ranks=[7,8,1,3,5,0,2,6,4] 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 10 monclient(hunting): picked mon.noname-i con 0x7f11d00a3860 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 10 monclient(hunting): picked mon.noname-e con 0x7f11d006c1d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 10 monclient(hunting): picked mon.noname-f con 0x7f11d00a4350 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d0061ae0 con 0x7f11d00a4350 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d00a0600 con 0x7f11d00a3860 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d00a0b30 con 0x7f11d006c1d0 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11d53c3640 10 monclient(hunting): authenticate will time out at 747.452291s 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11c67fc640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f11d006c1d0 0x7f11d013b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33646/0 (socket says 172.21.15.94:33646) 2024-09-16T10:29:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.112+0000 7f11c67fc640 1 -- 172.21.15.94:0/2252103330 learned_addr learned my addr 172.21.15.94:0/2252103330 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11c67fc640 1 -- 172.21.15.94:0/2252103330 >> v1:172.21.15.94:6793/0 conn(0x7f11d006c1d0 legacy=0x7f11d013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11cf7fe640 1 -- 172.21.15.94:0/2252103330 >> v1:172.21.15.152:6789/0 conn(0x7f11d00a4350 legacy=0x7f11d013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ceffd640 1 -- 172.21.15.94:0/2252103330 >> v1:172.21.15.152:6792/0 conn(0x7f11d00a3860 legacy=0x7f11d00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588394307 0 0) 0x7f11d00a0b30 con 0x7f11d006c1d0 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): my global_id is 4409 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11bc003520 con 0x7f11d006c1d0 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187726065 0 0) 0x7f11d0061ae0 con 0x7f11d00a4350 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): my global_id is 4375 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d00a0b30 con 0x7f11d00a4350 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662626913 0 0) 0x7f11d00a0600 con 0x7f11d00a3860 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient(hunting): my global_id is 4426 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d0061ae0 con 0x7f11d00a3860 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160770037 0 0) 0x7f11bc003520 con 0x7f11d006c1d0 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 >> v1:172.21.15.152:6792/0 conn(0x7f11d00a3860 legacy=0x7f11d00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 >> v1:172.21.15.152:6789/0 conn(0x7f11d00a4350 legacy=0x7f11d013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 monclient: found mon.noname-e 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d00a0e10 con 0x7f11d006c1d0 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11d53c3640 5 monclient: authenticate success, global_id 4409 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.114+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11b8001a60 con 0x7f11d006c1d0 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11ce7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11ce7fc640 10 monclient: dump: 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.128 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11b8001d70 con 0x7f11d006c1d0 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11ce7fc640 1 -- 172.21.15.94:0/2252103330 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11b8004be0 con 0x7f11d006c1d0 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 4 set_mon_vals no callback set 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11ce7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11ce7fc640 10 monclient: dump: 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 10 monclient: shutdown 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 1 -- 172.21.15.94:0/2252103330 >> v1:172.21.15.94:6793/0 conn(0x7f11d006c1d0 legacy=0x7f11d013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 1 -- 172.21.15.94:0/2252103330 shutdown_connections 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 1 -- 172.21.15.94:0/2252103330 >> 172.21.15.94:0/2252103330 conn(0x7f11d00a5960 msgr2=0x7f11d00a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 1 -- 172.21.15.94:0/2252103330 shutdown_connections 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.115+0000 7f11d53c3640 1 -- 172.21.15.94:0/2252103330 wait complete. 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.116+0000 7f11d53c3640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.116+0000 7f11d53c3640 10 monclient: monmap: 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.131 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.116+0000 7f11d53c3640 1 Processor -- start 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.116+0000 7f11d53c3640 1 -- start start 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.116+0000 7f11d53c3640 10 monclient: init 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 10 monclient: _add_conns ranks=[6,3,2,0,7,4,5,8,1] 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 10 monclient(hunting): picked mon.noname-d con 0x7f11d006c1d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 10 monclient(hunting): picked mon.noname-g con 0x7f11d00a3860 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 10 monclient(hunting): picked mon.noname-b con 0x7f11d00a4350 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d01aa8e0 con 0x7f11d00a4350 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d009f850 con 0x7f11d00a3860 2024-09-16T10:29:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d01abab0 con 0x7f11d006c1d0 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11d53c3640 10 monclient(hunting): authenticate will time out at 747.457374s 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11ceffd640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f11d006c1d0 0x7f11d01be6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60372/0 (socket says 172.21.15.94:60372) 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.117+0000 7f11ceffd640 1 -- 172.21.15.94:0/1605298379 learned_addr learned my addr 172.21.15.94:0/1605298379 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11ceffd640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6792/0 conn(0x7f11d006c1d0 legacy=0x7f11d01be6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11cf7fe640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6790/0 conn(0x7f11d00a4350 legacy=0x7f11d01d3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c67fc640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.152:6790/0 conn(0x7f11d00a3860 legacy=0x7f11d01cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891942253 0 0) 0x7f11d01abab0 con 0x7f11d006c1d0 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): my global_id is 4389 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b0003520 con 0x7f11d006c1d0 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180611411 0 0) 0x7f11d01aa8e0 con 0x7f11d00a4350 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): my global_id is 4358 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d01abab0 con 0x7f11d00a4350 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67022950 0 0) 0x7f11d009f850 con 0x7f11d00a3860 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 10 monclient(hunting): my global_id is 4530 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d01aa8e0 con 0x7f11d00a3860 2024-09-16T10:29:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2674802 0 0) 0x7f11b0003520 con 0x7f11d006c1d0 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d009f850 con 0x7f11d006c1d0 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.119+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3679253923 0 0) 0x7f11d01abab0 con 0x7f11d00a4350 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11b0003520 con 0x7f11d00a4350 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11c0003610 con 0x7f11d006c1d0 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): dump: 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11c80020e0 con 0x7f11d00a4350 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): dump: 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.138 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769865648 0 0) 0x7f11d01aa8e0 con 0x7f11d00a3860 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d01abab0 con 0x7f11d00a3860 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11b8001320 con 0x7f11d00a3860 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient(hunting): dump: 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.139 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.140 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2232080828 0 0) 0x7f11d009f850 con 0x7f11d006c1d0 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.152:6790/0 conn(0x7f11d00a3860 legacy=0x7f11d01cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6790/0 conn(0x7f11d00a4350 legacy=0x7f11d01d3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 monclient: found mon.d 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d01abd90 con 0x7f11d006c1d0 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11d53c3640 5 monclient: authenticate success, global_id 4389 2024-09-16T10:29:40.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11d53c3640 10 monclient: _renew_subs 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11d53c3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d01aa630 con 0x7f11d006c1d0 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11d53c3640 10 monclient: _renew_subs 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11d53c3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d01bedc0 con 0x7f11d006c1d0 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11c0003c70 con 0x7f11d006c1d0 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11c0004120 con 0x7f11d006c1d0 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11cdffb640 4 set_mon_vals no callback set 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.120+0000 7f11c7fff640 10 monclient: dump: 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.142 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.121+0000 7f11d53c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.121+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f11c00042b0 con 0x7f11d006c1d0 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.121+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f11b4006580 con 0x7f11b4002820 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.121+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3926+0+0 (unknown 1526538524 0 0) 0x7f11c0046150 con 0x7f11d006c1d0 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.121+0000 7f11cf7fe640 10 monclient: get_auth_request con 0x7f11b0039c40 auth_method 0 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c67fc640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6790/0 conn(0x7f11b4002820 legacy=0x7f11b4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565667008 0 0) 0x7f11b4006580 con 0x7f11b4002820 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 10 monclient(hunting): my global_id is 4367 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d009f850 con 0x7f11b4002820 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 426764306 0 0) 0x7f11d009f850 con 0x7f11b4002820 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11b4006580 con 0x7f11b4002820 2024-09-16T10:29:40.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11b8005050 con 0x7f11b4002820 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 10 monclient: dump: 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.145 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4000143722 0 0) 0x7f11b4006580 con 0x7f11b4002820 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.123+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b400a0d0 con 0x7f11b4002820 2024-09-16T10:29:40.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.124+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f11b8005370 con 0x7f11b4002820 2024-09-16T10:29:40.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.124+0000 7f11c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.124+0000 7f11c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:40.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.124+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6790/0 conn(0x7f11b4002820 legacy=0x7f11b4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.124+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6790/0 conn(0x7f11b4002820 legacy=0x7f11b4004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.130+0000 7f11d53c3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.130+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f11b4009d90 con 0x7f11b40090b0 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c67fc640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6790/0 conn(0x7f11b40090b0 legacy=0x7f11b4009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586994797 0 0) 0x7f11b4009d90 con 0x7f11b40090b0 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 10 monclient(hunting): my global_id is 4376 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b4006580 con 0x7f11b40090b0 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1836343509 0 0) 0x7f11b4006580 con 0x7f11b40090b0 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11b4009d90 con 0x7f11b40090b0 2024-09-16T10:29:40.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f11b8005aa0 con 0x7f11b40090b0 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 10 monclient: dump: 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1357602987 0 0) 0x7f11b4009d90 con 0x7f11b40090b0 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.132+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 --> v1:172.21.15.94:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f11b400c280 con 0x7f11b40090b0 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 <== mon.2 v1:172.21.15.94:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7784 (unknown 0 0 2141907728) 0x7f11b80080f0 con 0x7f11b40090b0 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6790/0 conn(0x7f11b40090b0 legacy=0x7f11b4009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11c7fff640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6790/0 conn(0x7f11b40090b0 legacy=0x7f11b4009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.152 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:29:40.152 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:40.152 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:40.152 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:29:40.152 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 8, 2024-09-16T10:29:40.152 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:29:40.152 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:29:40.152 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:29:40.152 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 24, 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.153 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.154 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:29:40.155 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:29:40.156 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.158 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:40.159 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.160 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.161 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:29:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": [ 2024-09-16T10:29:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:29:40.174 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:29:40.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.152:6800/2686360781 conn(0x7f11b0039c40 legacy=0x7f11b003c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11d53c3640 10 monclient: shutdown 2024-09-16T10:29:40.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11d53c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 >> v1:172.21.15.94:6792/0 conn(0x7f11d006c1d0 legacy=0x7f11d01be6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11cf7fe640 1 -- 172.21.15.94:0/1605298379 reap_dead start 2024-09-16T10:29:40.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11cf7fe640 1 -- 172.21.15.94:0/1605298379 reap_dead start 2024-09-16T10:29:40.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 shutdown_connections 2024-09-16T10:29:40.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 >> 172.21.15.94:0/1605298379 conn(0x7f11d00a5960 msgr2=0x7f11d00a7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 shutdown_connections 2024-09-16T10:29:40.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.133+0000 7f11d53c3640 1 -- 172.21.15.94:0/1605298379 wait complete. 2024-09-16T10:29:40.176 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:29:40.176 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-16T10:29:40.176 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:29:40.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.271+0000 7f80e7296640 10 monclient: get_monmap_and_config 2024-09-16T10:29:40.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.271+0000 7f80e7296640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.271+0000 7f80e7296640 10 monclient: monmap: 2024-09-16T10:29:40.273 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.273 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.273 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:40.272458+0000 2024-09-16T10:29:40.273 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:40.272458+0000 2024-09-16T10:29:40.273 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.273 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.274 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 1 Processor -- start 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 1 -- start start 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 10 monclient: init 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 10 monclient: _add_conns ranks=[1,6,8,5,0,3,7,4,2] 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 10 monclient(hunting): picked mon.noname-f con 0x7f80e00a3840 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 10 monclient(hunting): picked mon.noname-d con 0x7f80e00a17d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 10 monclient(hunting): picked mon.noname-e con 0x7f80e00a4330 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e0061ae0 con 0x7f80e00a3840 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e00a3280 con 0x7f80e00a17d0 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e006c7d0 con 0x7f80e00a4330 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.272+0000 7f80e7296640 10 monclient(hunting): authenticate will time out at 747.612923s 2024-09-16T10:29:40.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.273+0000 7f80e4831640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f80e00a17d0 0x7f80e013ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60376/0 (socket says 172.21.15.94:60376) 2024-09-16T10:29:40.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.273+0000 7f80e4831640 1 -- 172.21.15.94:0/284383609 learned_addr learned my addr 172.21.15.94:0/284383609 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.273+0000 7f80e5833640 1 --1- 172.21.15.94:0/284383609 >> v1:172.21.15.94:6793/0 conn(0x7f80e00a4330 0x7f80e013f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:33656/0 (socket says 172.21.15.94:33656) 2024-09-16T10:29:40.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80e4831640 1 -- 172.21.15.94:0/284383609 >> v1:172.21.15.94:6792/0 conn(0x7f80e00a17d0 legacy=0x7f80e013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80e5833640 1 -- 172.21.15.94:0/284383609 >> v1:172.21.15.94:6793/0 conn(0x7f80e00a4330 legacy=0x7f80e013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029383331 0 0) 0x7f80e00a3280 con 0x7f80e00a17d0 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient(hunting): my global_id is 4398 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80cc003520 con 0x7f80e00a17d0 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937696056 0 0) 0x7f80e006c7d0 con 0x7f80e00a4330 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient(hunting): my global_id is 4418 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e00a3280 con 0x7f80e00a4330 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80e5032640 1 -- 172.21.15.94:0/284383609 >> v1:172.21.15.152:6789/0 conn(0x7f80e00a3840 legacy=0x7f80e00a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2404073867 0 0) 0x7f80cc003520 con 0x7f80e00a17d0 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 >> v1:172.21.15.94:6793/0 conn(0x7f80e00a4330 legacy=0x7f80e013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 >> v1:172.21.15.152:6789/0 conn(0x7f80e00a3840 legacy=0x7f80e00a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 monclient: found mon.noname-d 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e00a0550 con 0x7f80e00a17d0 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f80d0001a60 con 0x7f80e00a17d0 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80e7296640 5 monclient: authenticate success, global_id 4398 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: dump: 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80e7296640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80d0001d70 con 0x7f80e00a17d0 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80e7296640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:40.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 1 -- 172.21.15.94:0/284383609 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f80d0004be0 con 0x7f80e00a17d0 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80e7296640 4 set_mon_vals no callback set 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.275+0000 7f80d7fff640 10 monclient: dump: 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.276+0000 7f80e7296640 10 monclient: shutdown 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.276+0000 7f80e7296640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.276+0000 7f80e7296640 1 -- 172.21.15.94:0/284383609 >> v1:172.21.15.94:6792/0 conn(0x7f80e00a17d0 legacy=0x7f80e013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.276+0000 7f80e7296640 1 -- 172.21.15.94:0/284383609 shutdown_connections 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.276+0000 7f80e7296640 1 -- 172.21.15.94:0/284383609 >> 172.21.15.94:0/284383609 conn(0x7f80e00a97b0 msgr2=0x7f80e00a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.276+0000 7f80e7296640 1 -- 172.21.15.94:0/284383609 shutdown_connections 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.276+0000 7f80e7296640 1 -- 172.21.15.94:0/284383609 wait complete. 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.276+0000 7f80e7296640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.276+0000 7f80e7296640 10 monclient: monmap: 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 1 Processor -- start 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 1 -- start start 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 10 monclient: init 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 10 monclient: _add_conns ranks=[7,3,5,2,1,6,8,0,4] 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 10 monclient(hunting): picked mon.noname-i con 0x7f80e00a3840 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 10 monclient(hunting): picked mon.noname-g con 0x7f80e00a4330 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 10 monclient(hunting): picked mon.noname-h con 0x7f80e0137080 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e009f340 con 0x7f80e00a4330 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e009f010 con 0x7f80e0137080 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e0126da0 con 0x7f80e00a3840 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e7296640 10 monclient(hunting): authenticate will time out at 747.617670s 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e4831640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7f80e00a4330 0x7f80e0136970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.94:46598/0 (socket says 172.21.15.94:46598) 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.277+0000 7f80e4831640 1 -- 172.21.15.94:0/2038774018 learned_addr learned my addr 172.21.15.94:0/2038774018 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.279+0000 7f80e4831640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.152:6790/0 conn(0x7f80e00a4330 legacy=0x7f80e0136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80e5833640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.152:6791/0 conn(0x7f80e0137080 legacy=0x7f80e01d2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80e5032640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.152:6792/0 conn(0x7f80e00a3840 legacy=0x7f80e01caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151879701 0 0) 0x7f80e009f340 con 0x7f80e00a4330 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): my global_id is 4539 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c4003520 con 0x7f80e00a4330 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068046483 0 0) 0x7f80e009f010 con 0x7f80e0137080 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): my global_id is 4451 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e009f340 con 0x7f80e0137080 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086047280 0 0) 0x7f80e0126da0 con 0x7f80e00a3840 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): my global_id is 4435 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e009f010 con 0x7f80e00a3840 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2580296004 0 0) 0x7f80c4003520 con 0x7f80e00a4330 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80e0126da0 con 0x7f80e00a4330 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387805812 0 0) 0x7f80e009f010 con 0x7f80e00a3840 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80c4003520 con 0x7f80e00a3840 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f80d0004b20 con 0x7f80e00a4330 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): dump: 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 272337866 0 0) 0x7f80e009f340 con 0x7f80e0137080 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80e009f010 con 0x7f80e0137080 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f80dc001cd0 con 0x7f80e00a3840 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): dump: 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f80d8002000 con 0x7f80e0137080 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient(hunting): dump: 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1720456108 0 0) 0x7f80e0126da0 con 0x7f80e00a4330 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.152:6792/0 conn(0x7f80e00a3840 legacy=0x7f80e01caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.152:6791/0 conn(0x7f80e0137080 legacy=0x7f80e01d2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 monclient: found mon.g 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e0127080 con 0x7f80e00a4330 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80e7296640 5 monclient: authenticate success, global_id 4539 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80e7296640 10 monclient: _renew_subs 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80e7296640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e01256e0 con 0x7f80e00a4330 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80d0001a40 con 0x7f80e00a4330 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f80d0005550 con 0x7f80e00a4330 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d5ffb640 10 monclient: dump: 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.280+0000 7f80d77fe640 4 set_mon_vals no callback set 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.281+0000 7f80e7296640 10 monclient: _renew_subs 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.281+0000 7f80e7296640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.281+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e01d4680 con 0x7f80e00a4330 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.281+0000 7f80e7296640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.281+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f80e0067530 con 0x7f80e00a17d0 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.281+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f80d00105b0 con 0x7f80e00a4330 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.281+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3926+0+0 (unknown 1526538524 0 0) 0x7f80d00462b0 con 0x7f80e00a4330 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.282+0000 7f80e5833640 10 monclient: get_auth_request con 0x7f80c4039b70 auth_method 0 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80e5032640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.94:6791/0 conn(0x7f80e00a17d0 legacy=0x7f80e011f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653116771 0 0) 0x7f80e0067530 con 0x7f80e00a17d0 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 10 monclient(hunting): my global_id is 4423 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e0126da0 con 0x7f80e00a17d0 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2389674998 0 0) 0x7f80e0126da0 con 0x7f80e00a17d0 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80e0067530 con 0x7f80e00a17d0 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f80dc003f40 con 0x7f80e00a17d0 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 10 monclient: dump: 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1244515384 0 0) 0x7f80e0067530 con 0x7f80e00a17d0 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.94:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80e006b980 con 0x7f80e00a17d0 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.284+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f80dc004480 con 0x7f80e00a17d0 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.285+0000 7f80d5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.285+0000 7f80d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.285+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.94:6791/0 conn(0x7f80e00a17d0 legacy=0x7f80e011f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.285+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.94:6791/0 conn(0x7f80e00a17d0 legacy=0x7f80e011f490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.290+0000 7f80e7296640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.291+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f80e00665f0 con 0x7f80e002c270 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80e5032640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.94:6791/0 conn(0x7f80e002c270 legacy=0x7f80e011fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674877731 0 0) 0x7f80e00665f0 con 0x7f80e002c270 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 10 monclient(hunting): my global_id is 4432 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e0067530 con 0x7f80e002c270 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1898669967 0 0) 0x7f80e0067530 con 0x7f80e002c270 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80e00665f0 con 0x7f80e002c270 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f80dc005cf0 con 0x7f80e002c270 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 10 monclient: dump: 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2321809556 0 0) 0x7f80e00665f0 con 0x7f80e002c270 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 --> v1:172.21.15.94:6791/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f80e009eba0 con 0x7f80e002c270 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 <== mon.4 v1:172.21.15.94:6791/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7452 (unknown 0 0 99014646) 0x7f80dc008070 con 0x7f80e002c270 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.94:6791/0 conn(0x7f80e002c270 legacy=0x7f80e011fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.293+0000 7f80d5ffb640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.94:6791/0 conn(0x7f80e002c270 legacy=0x7f80e011fba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 8, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 24, 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.152:6800/2686360781 conn(0x7f80c4039b70 legacy=0x7f80c403c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e7296640 10 monclient: shutdown 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e7296640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 >> v1:172.21.15.152:6790/0 conn(0x7f80e00a4330 legacy=0x7f80e0136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e5833640 1 -- 172.21.15.94:0/2038774018 reap_dead start 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e5833640 1 -- 172.21.15.94:0/2038774018 reap_dead start 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 shutdown_connections 2024-09-16T10:29:40.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 >> 172.21.15.94:0/2038774018 conn(0x7f80e00a97b0 msgr2=0x7f80e009ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 shutdown_connections 2024-09-16T10:29:40.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.294+0000 7f80e7296640 1 -- 172.21.15.94:0/2038774018 wait complete. 2024-09-16T10:29:40.336 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:29:40.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-16T10:29:40.336 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-16T10:29:40.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.433+0000 7f9d125fb640 10 monclient: get_monmap_and_config 2024-09-16T10:29:40.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.433+0000 7f9d125fb640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.433+0000 7f9d125fb640 10 monclient: monmap: 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:40.433926+0000 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:40.433926+0000 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.435 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.436 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.436 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.436 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.436 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 1 Processor -- start 2024-09-16T10:29:40.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 1 -- start start 2024-09-16T10:29:40.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 10 monclient: init 2024-09-16T10:29:40.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 10 monclient: _add_conns ranks=[7,6,0,8,1,3,4,2,5] 2024-09-16T10:29:40.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 10 monclient(hunting): picked mon.noname-i con 0x7f9d0c0a38d0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:40.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 10 monclient(hunting): picked mon.noname-d con 0x7f9d0c06c670 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:40.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 10 monclient(hunting): picked mon.noname-a con 0x7f9d0c0a43c0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:40.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c061ae0 con 0x7f9d0c0a43c0 2024-09-16T10:29:40.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c0a04d0 con 0x7f9d0c06c670 2024-09-16T10:29:40.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c0a3250 con 0x7f9d0c0a38d0 2024-09-16T10:29:40.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d125fb640 10 monclient(hunting): authenticate will time out at 747.774600s 2024-09-16T10:29:40.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d03fff640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f9d0c06c670 0x7f9d0c13bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60384/0 (socket says 172.21.15.94:60384) 2024-09-16T10:29:40.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d03fff640 1 -- 172.21.15.94:0/3487391593 learned_addr learned my addr 172.21.15.94:0/3487391593 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.434+0000 7f9d10b98640 1 --1- 172.21.15.94:0/3487391593 >> v1:172.21.15.94:6789/0 conn(0x7f9d0c0a43c0 0x7f9d0c13f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41594/0 (socket says 172.21.15.94:41594) 2024-09-16T10:29:40.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.436+0000 7f9d03fff640 1 -- 172.21.15.94:0/3487391593 >> v1:172.21.15.94:6792/0 conn(0x7f9d0c06c670 legacy=0x7f9d0c13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.436+0000 7f9d10b98640 1 -- 172.21.15.94:0/3487391593 >> v1:172.21.15.94:6789/0 conn(0x7f9d0c0a43c0 legacy=0x7f9d0c13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857737691 0 0) 0x7f9d0c0a04d0 con 0x7f9d0c06c670 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient(hunting): my global_id is 4407 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf8003520 con 0x7f9d0c06c670 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647061444 0 0) 0x7f9d0c061ae0 con 0x7f9d0c0a43c0 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient(hunting): my global_id is 4437 2024-09-16T10:29:40.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0c0a04d0 con 0x7f9d0c0a43c0 2024-09-16T10:29:40.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0bfff640 1 -- 172.21.15.94:0/3487391593 >> v1:172.21.15.152:6792/0 conn(0x7f9d0c0a38d0 legacy=0x7f9d0c0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2589275361 0 0) 0x7f9cf8003520 con 0x7f9d0c06c670 2024-09-16T10:29:40.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 >> v1:172.21.15.152:6792/0 conn(0x7f9d0c0a38d0 legacy=0x7f9d0c0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 >> v1:172.21.15.94:6789/0 conn(0x7f9d0c0a43c0 legacy=0x7f9d0c13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 monclient: found mon.noname-d 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c0a0a00 con 0x7f9d0c06c670 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9cf4001a60 con 0x7f9d0c06c670 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: dump: 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.445 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d125fb640 5 monclient: authenticate success, global_id 4407 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d125fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cf4001d70 con 0x7f9d0c06c670 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 1 -- 172.21.15.94:0/3487391593 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9cf4004be0 con 0x7f9d0c06c670 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d125fb640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d125fb640 4 set_mon_vals no callback set 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d0b7fe640 10 monclient: dump: 2024-09-16T10:29:40.446 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d125fb640 10 monclient: shutdown 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d125fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.437+0000 7f9d125fb640 1 -- 172.21.15.94:0/3487391593 >> v1:172.21.15.94:6792/0 conn(0x7f9d0c06c670 legacy=0x7f9d0c13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.438+0000 7f9d125fb640 1 -- 172.21.15.94:0/3487391593 shutdown_connections 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.438+0000 7f9d125fb640 1 -- 172.21.15.94:0/3487391593 >> 172.21.15.94:0/3487391593 conn(0x7f9d0c0a59d0 msgr2=0x7f9d0c0a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.438+0000 7f9d125fb640 1 -- 172.21.15.94:0/3487391593 shutdown_connections 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.438+0000 7f9d125fb640 1 -- 172.21.15.94:0/3487391593 wait complete. 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.438+0000 7f9d125fb640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.438+0000 7f9d125fb640 10 monclient: monmap: 2024-09-16T10:29:40.448 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.438+0000 7f9d125fb640 1 Processor -- start 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.438+0000 7f9d125fb640 1 -- start start 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.438+0000 7f9d125fb640 10 monclient: init 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 10 monclient: _add_conns ranks=[4,2,5,7,8,0,1,6,3] 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 10 monclient(hunting): picked mon.noname-c con 0x7f9d0c06c670 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 10 monclient(hunting): picked mon.noname-b con 0x7f9d0c0a38d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 10 monclient(hunting): picked mon.noname-h con 0x7f9d0c0a43c0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c09edf0 con 0x7f9d0c0a38d0 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c06b830 con 0x7f9d0c06c670 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c1266f0 con 0x7f9d0c0a43c0 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d125fb640 10 monclient(hunting): authenticate will time out at 747.779435s 2024-09-16T10:29:40.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d0bfff640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f9d0c06c670 0x7f9d0c136140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43604/0 (socket says 172.21.15.94:43604) 2024-09-16T10:29:40.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.439+0000 7f9d0bfff640 1 -- 172.21.15.94:0/1968371508 learned_addr learned my addr 172.21.15.94:0/1968371508 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.441+0000 7f9d0bfff640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6791/0 conn(0x7f9d0c06c670 legacy=0x7f9d0c136140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.441+0000 7f9d03fff640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6790/0 conn(0x7f9d0c0a38d0 legacy=0x7f9d0c1cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.441+0000 7f9d10b98640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.152:6791/0 conn(0x7f9d0c0a43c0 legacy=0x7f9d0c1d3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.441+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 DispatchQueue::entry inject delay of 0.002000 2024-09-16T10:29:40.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.443+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126654974 0 0) 0x7f9d0c06b830 con 0x7f9d0c06c670 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.443+0000 7f9d097fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): my global_id is 4441 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cec003520 con 0x7f9d0c06c670 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524579301 0 0) 0x7f9d0c09edf0 con 0x7f9d0c0a38d0 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): my global_id is 4385 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0c06b830 con 0x7f9d0c0a38d0 2024-09-16T10:29:40.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762424414 0 0) 0x7f9d0c1266f0 con 0x7f9d0c0a43c0 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): my global_id is 4460 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0c09edf0 con 0x7f9d0c0a43c0 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1360194324 0 0) 0x7f9cec003520 con 0x7f9d0c06c670 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d0c1266f0 con 0x7f9d0c06c670 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9cfc001d30 con 0x7f9d0c06c670 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): dump: 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.459 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.460 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3056245153 0 0) 0x7f9d0c06b830 con 0x7f9d0c0a38d0 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cec003520 con 0x7f9d0c0a38d0 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9cf4004b20 con 0x7f9d0c0a38d0 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): dump: 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.461 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4063507580 0 0) 0x7f9d0c09edf0 con 0x7f9d0c0a43c0 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d0c06b830 con 0x7f9d0c0a43c0 2024-09-16T10:29:40.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9d04002010 con 0x7f9d0c0a43c0 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient(hunting): dump: 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.463 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4106806474 0 0) 0x7f9d0c1266f0 con 0x7f9d0c06c670 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.152:6791/0 conn(0x7f9d0c0a43c0 legacy=0x7f9d0c1d3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6790/0 conn(0x7f9d0c0a38d0 legacy=0x7f9d0c1cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 monclient: found mon.c 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c136f40 con 0x7f9d0c06c670 2024-09-16T10:29:40.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d125fb640 5 monclient: authenticate success, global_id 4441 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d125fb640 10 monclient: _renew_subs 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d125fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c124bc0 con 0x7f9d0c06c670 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d125fb640 10 monclient: _renew_subs 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d125fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c09f120 con 0x7f9d0c06c670 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cfc003b70 con 0x7f9d0c06c670 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9cfc004020 con 0x7f9d0c06c670 2024-09-16T10:29:40.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d0affd640 4 set_mon_vals no callback set 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.444+0000 7f9d097fa640 10 monclient: dump: 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.445+0000 7f9d125fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.445+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9cf0006580 con 0x7f9cf0002820 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.445+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f9cfc004180 con 0x7f9d0c06c670 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.445+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3926+0+0 (unknown 1526538524 0 0) 0x7f9cfc046000 con 0x7f9d0c06c670 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.445+0000 7f9d10b98640 10 monclient: get_auth_request con 0x7f9cec039d20 auth_method 0 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d03fff640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6792/0 conn(0x7f9cf0002820 legacy=0x7f9cf0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388988770 0 0) 0x7f9cf0006580 con 0x7f9cf0002820 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 10 monclient(hunting): my global_id is 4416 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0c1266f0 con 0x7f9cf0002820 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 228182560 0 0) 0x7f9d0c1266f0 con 0x7f9cf0002820 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cf0006580 con 0x7f9cf0002820 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9cf4001320 con 0x7f9cf0002820 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.447+0000 7f9d097fa640 10 monclient: dump: 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.468 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.469 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.469 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.469 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.469 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.469 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.469 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.469 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.470 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.470 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.470 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.470 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.470 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.470 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.448+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4127938705 0 0) 0x7f9cf0006580 con 0x7f9cf0002820 2024-09-16T10:29:40.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.448+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9cf000a0d0 con 0x7f9cf0002820 2024-09-16T10:29:40.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.448+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9cf4005870 con 0x7f9cf0002820 2024-09-16T10:29:40.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.448+0000 7f9d097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.448+0000 7f9d097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:40.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.448+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6792/0 conn(0x7f9cf0002820 legacy=0x7f9cf0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.448+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6792/0 conn(0x7f9cf0002820 legacy=0x7f9cf0004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.454+0000 7f9d125fb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:29:40.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.454+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9cf0009d90 con 0x7f9cf00090b0 2024-09-16T10:29:40.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d03fff640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6792/0 conn(0x7f9cf00090b0 legacy=0x7f9cf0009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948058148 0 0) 0x7f9cf0009d90 con 0x7f9cf00090b0 2024-09-16T10:29:40.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 10 monclient(hunting): my global_id is 4425 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf0006580 con 0x7f9cf00090b0 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3339333948 0 0) 0x7f9cf0006580 con 0x7f9cf00090b0 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cf0009d90 con 0x7f9cf00090b0 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9cf4005fb0 con 0x7f9cf00090b0 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.456+0000 7f9d097fa640 10 monclient: dump: 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.473 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.474 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.474 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.474 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.474 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.474 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.475 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.475 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.475 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.475 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2994939430 0 0) 0x7f9cf0009d90 con 0x7f9cf00090b0 2024-09-16T10:29:40.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 --> v1:172.21.15.94:6792/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9cf000c280 con 0x7f9cf00090b0 2024-09-16T10:29:40.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 <== mon.6 v1:172.21.15.94:6792/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7452 (unknown 0 0 3436787584) 0x7f9cf40082d0 con 0x7f9cf00090b0 2024-09-16T10:29:40.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:29:40.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:40.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6792/0 conn(0x7f9cf00090b0 legacy=0x7f9cf0009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d097fa640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6792/0 conn(0x7f9cf00090b0 legacy=0x7f9cf0009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.476 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:29:40.476 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:40.476 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:40.476 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:29:40.476 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 8, 2024-09-16T10:29:40.476 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:29:40.477 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 24, 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.478 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.479 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.480 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.481 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:40.482 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.483 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:29:40.484 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:40.485 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.486 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.487 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.490 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.491 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.492 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.493 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.495 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:29:40.496 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:29:40.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.152:6800/2686360781 conn(0x7f9cec039d20 legacy=0x7f9cec03c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d125fb640 10 monclient: shutdown 2024-09-16T10:29:40.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d125fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 >> v1:172.21.15.94:6791/0 conn(0x7f9d0c06c670 legacy=0x7f9d0c136140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d10b98640 1 -- 172.21.15.94:0/1968371508 reap_dead start 2024-09-16T10:29:40.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d10b98640 1 -- 172.21.15.94:0/1968371508 reap_dead start 2024-09-16T10:29:40.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 shutdown_connections 2024-09-16T10:29:40.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 >> 172.21.15.94:0/1968371508 conn(0x7f9d0c0a59d0 msgr2=0x7f9d0c09f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 shutdown_connections 2024-09-16T10:29:40.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.457+0000 7f9d125fb640 1 -- 172.21.15.94:0/1968371508 wait complete. 2024-09-16T10:29:40.500 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:29:40.500 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-16T10:29:40.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-16T10:29:40.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.596+0000 7f8cdf108640 10 monclient: get_monmap_and_config 2024-09-16T10:29:40.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.596+0000 7f8cdf108640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.597+0000 7f8cdf108640 10 monclient: monmap: 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:40.597780+0000 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:40.597780+0000 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.606 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.597+0000 7f8cdf108640 1 Processor -- start 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.597+0000 7f8cdf108640 1 -- start start 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.597+0000 7f8cdf108640 10 monclient: init 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 10 monclient: _add_conns ranks=[2,4,3,0,8,6,1,7,5] 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 10 monclient(hunting): picked mon.noname-b con 0x7f8cd80a3860 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 10 monclient(hunting): picked mon.noname-c con 0x7f8cd806c1d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 10 monclient(hunting): picked mon.noname-g con 0x7f8cd80a4350 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd8061ae0 con 0x7f8cd80a3860 2024-09-16T10:29:40.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd80a0600 con 0x7f8cd80a4350 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd80a0b30 con 0x7f8cd806c1d0 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdf108640 10 monclient(hunting): authenticate will time out at 747.938287s 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdcea4640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f8cd80a3860 0x7f8cd80a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48512/0 (socket says 172.21.15.94:48512) 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.598+0000 7f8cdcea4640 1 -- 172.21.15.94:0/3536862422 learned_addr learned my addr 172.21.15.94:0/3536862422 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8cdcea4640 1 -- 172.21.15.94:0/3536862422 >> v1:172.21.15.94:6790/0 conn(0x7f8cd80a3860 legacy=0x7f8cd80a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccffff640 1 -- 172.21.15.94:0/3536862422 >> v1:172.21.15.94:6791/0 conn(0x7f8cd806c1d0 legacy=0x7f8cd813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8cdd6a5640 1 -- 172.21.15.94:0/3536862422 >> v1:172.21.15.152:6790/0 conn(0x7f8cd80a4350 legacy=0x7f8cd813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837533959 0 0) 0x7f8cd80a0b30 con 0x7f8cd806c1d0 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): my global_id is 4450 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc0003520 con 0x7f8cd806c1d0 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268752552 0 0) 0x7f8cd8061ae0 con 0x7f8cd80a3860 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): my global_id is 4394 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd80a0b30 con 0x7f8cd80a3860 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234836219 0 0) 0x7f8cd80a0600 con 0x7f8cd80a4350 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient(hunting): my global_id is 4548 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd8061ae0 con 0x7f8cd80a4350 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3408443801 0 0) 0x7f8cc0003520 con 0x7f8cd806c1d0 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 >> v1:172.21.15.152:6790/0 conn(0x7f8cd80a4350 legacy=0x7f8cd813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 >> v1:172.21.15.94:6790/0 conn(0x7f8cd80a3860 legacy=0x7f8cd80a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 monclient: found mon.noname-c 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd80a0e10 con 0x7f8cd806c1d0 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 5 monclient: authenticate success, global_id 4450 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.600+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8cc8001ab0 con 0x7f8cd806c1d0 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8ccf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8ccf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8ccf7fe640 10 monclient: dump: 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cc8003610 con 0x7f8cd806c1d0 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8ccf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8ccf7fe640 1 -- 172.21.15.94:0/3536862422 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8cc8003b00 con 0x7f8cd806c1d0 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 4 set_mon_vals no callback set 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8ccf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8ccf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8ccf7fe640 10 monclient: dump: 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.613 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 10 monclient: shutdown 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 1 -- 172.21.15.94:0/3536862422 >> v1:172.21.15.94:6791/0 conn(0x7f8cd806c1d0 legacy=0x7f8cd813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 1 -- 172.21.15.94:0/3536862422 shutdown_connections 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 1 -- 172.21.15.94:0/3536862422 >> 172.21.15.94:0/3536862422 conn(0x7f8cd80a5960 msgr2=0x7f8cd80a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 1 -- 172.21.15.94:0/3536862422 shutdown_connections 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.601+0000 7f8cdf108640 1 -- 172.21.15.94:0/3536862422 wait complete. 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.602+0000 7f8cdf108640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.602+0000 7f8cdf108640 10 monclient: monmap: 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.602+0000 7f8cdf108640 1 Processor -- start 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.602+0000 7f8cdf108640 1 -- start start 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.602+0000 7f8cdf108640 10 monclient: init 2024-09-16T10:29:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 10 monclient: _add_conns ranks=[7,6,4,5,0,1,3,8,2] 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 10 monclient(hunting): picked mon.noname-i con 0x7f8cd806c1d0 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 10 monclient(hunting): picked mon.noname-d con 0x7f8cd80a3860 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 10 monclient(hunting): picked mon.noname-c con 0x7f8cd80a4350 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd809f650 con 0x7f8cd80a4350 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd806ba50 con 0x7f8cd80a3860 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd81cb630 con 0x7f8cd806c1d0 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdf108640 10 monclient(hunting): authenticate will time out at 747.943203s 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8ccffff640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f8cd80a3860 0x7f8cd81cfa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60402/0 (socket says 172.21.15.94:60402) 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8ccffff640 1 -- 172.21.15.94:0/4058686111 learned_addr learned my addr 172.21.15.94:0/4058686111 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.603+0000 7f8cdd6a5640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f8cd80a4350 0x7f8cd81d3160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43614/0 (socket says 172.21.15.94:43614) 2024-09-16T10:29:40.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccffff640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6792/0 conn(0x7f8cd80a3860 legacy=0x7f8cd81cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8cdd6a5640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6791/0 conn(0x7f8cd80a4350 legacy=0x7f8cd81d3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8cdcea4640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.152:6792/0 conn(0x7f8cd806c1d0 legacy=0x7f8cd81be470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112980575 0 0) 0x7f8cd809f650 con 0x7f8cd80a4350 2024-09-16T10:29:40.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): my global_id is 4459 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc4003520 con 0x7f8cd80a4350 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301387415 0 0) 0x7f8cd806ba50 con 0x7f8cd80a3860 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): my global_id is 4434 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd809f650 con 0x7f8cd80a3860 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525239220 0 0) 0x7f8cd81cb630 con 0x7f8cd806c1d0 2024-09-16T10:29:40.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): my global_id is 4453 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd806ba50 con 0x7f8cd806c1d0 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1129137405 0 0) 0x7f8cd809f650 con 0x7f8cd80a3860 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cd81cb630 con 0x7f8cd80a3860 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2241905335 0 0) 0x7f8cc4003520 con 0x7f8cd80a4350 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cd809f650 con 0x7f8cd80a4350 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8cc8003ac0 con 0x7f8cd80a3860 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): dump: 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.625 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8cd0001ff0 con 0x7f8cd80a4350 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): dump: 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.627 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4268580810 0 0) 0x7f8cd806ba50 con 0x7f8cd806c1d0 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cc4003520 con 0x7f8cd806c1d0 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8cd4001d00 con 0x7f8cd806c1d0 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.605+0000 7f8ccd7fa640 10 monclient(hunting): dump: 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.629 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3918181276 0 0) 0x7f8cd81cb630 con 0x7f8cd80a3860 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.152:6792/0 conn(0x7f8cd806c1d0 legacy=0x7f8cd81be470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6791/0 conn(0x7f8cd80a4350 legacy=0x7f8cd81d3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 1 monclient: found mon.d 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd81aa8c0 con 0x7f8cd80a3860 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cdf108640 5 monclient: authenticate success, global_id 4434 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cdf108640 10 monclient: _renew_subs 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cdf108640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd809f100 con 0x7f8cd80a3860 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cdf108640 10 monclient: _renew_subs 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cdf108640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd81ab590 con 0x7f8cd80a3860 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cc80040b0 con 0x7f8cd80a3860 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8cc8001d80 con 0x7f8cd80a3860 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cceffd640 4 set_mon_vals no callback set 2024-09-16T10:29:40.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8ccd7fa640 10 monclient: dump: 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.632 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cdf108640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.606+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8cb8006580 con 0x7f8cb8002820 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.607+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f8cc80106c0 con 0x7f8cd80a3860 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.607+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3926+0+0 (unknown 1526538524 0 0) 0x7f8cc8046300 con 0x7f8cd80a3860 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.607+0000 7f8cdd6a5640 10 monclient: get_auth_request con 0x7f8cc4039c20 auth_method 0 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8cdcea4640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6793/0 conn(0x7f8cb8002820 legacy=0x7f8cb8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842165816 0 0) 0x7f8cb8006580 con 0x7f8cb8002820 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 10 monclient(hunting): my global_id is 4427 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd81cb630 con 0x7f8cb8002820 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3249895360 0 0) 0x7f8cd81cb630 con 0x7f8cb8002820 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb8006580 con 0x7f8cb8002820 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8cd4004d90 con 0x7f8cb8002820 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 10 monclient: dump: 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.635 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3595505901 0 0) 0x7f8cb8006580 con 0x7f8cb8002820 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8cb800a0d0 con 0x7f8cb8002820 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8cd40071a0 con 0x7f8cb8002820 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6793/0 conn(0x7f8cb8002820 legacy=0x7f8cb8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.609+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6793/0 conn(0x7f8cb8002820 legacy=0x7f8cb8004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.616+0000 7f8cdf108640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.616+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8cb8009d90 con 0x7f8cb80090b0 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8cdcea4640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6793/0 conn(0x7f8cb80090b0 legacy=0x7f8cb8009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158407015 0 0) 0x7f8cb8009d90 con 0x7f8cb80090b0 2024-09-16T10:29:40.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 10 monclient(hunting): my global_id is 4436 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb8006580 con 0x7f8cb80090b0 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1740819150 0 0) 0x7f8cb8006580 con 0x7f8cb80090b0 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb8009d90 con 0x7f8cb80090b0 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8cd4005960 con 0x7f8cb80090b0 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.618+0000 7f8ccd7fa640 10 monclient: dump: 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.637 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1206288268 0 0) 0x7f8cb8009d90 con 0x7f8cb80090b0 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 --> v1:172.21.15.94:6793/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8cb800c280 con 0x7f8cb80090b0 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 <== mon.8 v1:172.21.15.94:6793/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7618 (unknown 0 0 4257498686) 0x7f8cd4006f20 con 0x7f8cb80090b0 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8ccd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8ccd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6793/0 conn(0x7f8cb80090b0 legacy=0x7f8cb8009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8ccd7fa640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6793/0 conn(0x7f8cb80090b0 legacy=0x7f8cb8009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:29:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 8, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 24, 2024-09-16T10:29:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:29:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:29:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:29:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:29:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:29:40.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.660 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.660 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.660 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.660 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.660 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:29:40.660 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:29:40.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8cc4039c20 legacy=0x7f8cc403c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdf108640 10 monclient: shutdown 2024-09-16T10:29:40.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdf108640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 >> v1:172.21.15.94:6792/0 conn(0x7f8cd80a3860 legacy=0x7f8cd81cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdd6a5640 1 -- 172.21.15.94:0/4058686111 reap_dead start 2024-09-16T10:29:40.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdd6a5640 1 -- 172.21.15.94:0/4058686111 reap_dead start 2024-09-16T10:29:40.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 shutdown_connections 2024-09-16T10:29:40.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 >> 172.21.15.94:0/4058686111 conn(0x7f8cd80a5960 msgr2=0x7f8cd80a77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 shutdown_connections 2024-09-16T10:29:40.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.619+0000 7f8cdf108640 1 -- 172.21.15.94:0/4058686111 wait complete. 2024-09-16T10:29:40.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:29:40.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-16T10:29:40.663 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.759+0000 7fe83683a640 10 monclient: get_monmap_and_config 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.759+0000 7fe83683a640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.759+0000 7fe83683a640 10 monclient: monmap: 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:40.760306+0000 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:40.760306+0000 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.761 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.762 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.762 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.762 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.762 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.762 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.762 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.762 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.762 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.762 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.760+0000 7fe83683a640 1 Processor -- start 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.760+0000 7fe83683a640 1 -- start start 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.760+0000 7fe83683a640 10 monclient: init 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 10 monclient: _add_conns ranks=[2,5,1,7,6,0,4,3,8] 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 10 monclient(hunting): picked mon.noname-b con 0x7fe8300a3860 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 10 monclient(hunting): picked mon.noname-h con 0x7fe83006c1d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 10 monclient(hunting): picked mon.noname-f con 0x7fe8300a4350 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe830061ae0 con 0x7fe8300a4350 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8300a0600 con 0x7fe8300a3860 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe82ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fe8300a3860 0x7fe8300a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48516/0 (socket says 172.21.15.94:48516) 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe82ffff640 1 -- 172.21.15.94:0/1813552014 learned_addr learned my addr 172.21.15.94:0/1813552014 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 1 -- 172.21.15.94:0/1813552014 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8300a0b30 con 0x7fe83006c1d0 2024-09-16T10:29:40.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.761+0000 7fe83683a640 10 monclient(hunting): authenticate will time out at 748.101298s 2024-09-16T10:29:40.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.763+0000 7fe827fff640 1 -- 172.21.15.94:0/1813552014 >> v1:172.21.15.152:6791/0 conn(0x7fe83006c1d0 legacy=0x7fe83013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.763+0000 7fe834dd7640 1 -- 172.21.15.94:0/1813552014 >> v1:172.21.15.152:6789/0 conn(0x7fe8300a4350 legacy=0x7fe83013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.763+0000 7fe82ffff640 1 -- 172.21.15.94:0/1813552014 >> v1:172.21.15.94:6790/0 conn(0x7fe8300a3860 legacy=0x7fe8300a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 1 -- 172.21.15.94:0/1813552014 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262328122 0 0) 0x7fe8300a0600 con 0x7fe8300a3860 2024-09-16T10:29:40.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 10 monclient(hunting): my global_id is 4403 2024-09-16T10:29:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 1 -- 172.21.15.94:0/1813552014 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe81c003520 con 0x7fe8300a3860 2024-09-16T10:29:40.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 1 -- 172.21.15.94:0/1813552014 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283698301 0 0) 0x7fe81c003520 con 0x7fe8300a3860 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 1 -- 172.21.15.94:0/1813552014 >> v1:172.21.15.152:6791/0 conn(0x7fe83006c1d0 legacy=0x7fe83013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 1 -- 172.21.15.94:0/1813552014 >> v1:172.21.15.152:6789/0 conn(0x7fe8300a4350 legacy=0x7fe83013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 1 monclient: found mon.noname-b 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 1 -- 172.21.15.94:0/1813552014 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8300a0e10 con 0x7fe8300a3860 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 1 -- 172.21.15.94:0/1813552014 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe820001a60 con 0x7fe8300a3860 2024-09-16T10:29:40.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.765+0000 7fe82f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe82f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe82f7fe640 10 monclient: dump: 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.770 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe83683a640 5 monclient: authenticate success, global_id 4403 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe83683a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe82f7fe640 1 -- 172.21.15.94:0/1813552014 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe820001d70 con 0x7fe8300a3860 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe82f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe83683a640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:40.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe82f7fe640 1 -- 172.21.15.94:0/1813552014 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe820004be0 con 0x7fe8300a3860 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe83683a640 4 set_mon_vals no callback set 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe82f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe82f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.766+0000 7fe82f7fe640 10 monclient: dump: 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.772 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.773 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.773 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.773 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.773 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.773 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.773 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.773 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.773 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.768+0000 7fe83683a640 10 monclient: shutdown 2024-09-16T10:29:40.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.768+0000 7fe83683a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.768+0000 7fe83683a640 1 -- 172.21.15.94:0/1813552014 >> v1:172.21.15.94:6790/0 conn(0x7fe8300a3860 legacy=0x7fe8300a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.769+0000 7fe83683a640 1 -- 172.21.15.94:0/1813552014 shutdown_connections 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.769+0000 7fe83683a640 1 -- 172.21.15.94:0/1813552014 >> 172.21.15.94:0/1813552014 conn(0x7fe8300a5960 msgr2=0x7fe8300a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.771+0000 7fe83683a640 1 -- 172.21.15.94:0/1813552014 shutdown_connections 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.771+0000 7fe83683a640 1 -- 172.21.15.94:0/1813552014 wait complete. 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.772+0000 7fe83683a640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.772+0000 7fe83683a640 10 monclient: monmap: 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:40.776 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.777 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.772+0000 7fe83683a640 1 Processor -- start 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.772+0000 7fe83683a640 1 -- start start 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.772+0000 7fe83683a640 10 monclient: init 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 10 monclient: _add_conns ranks=[3,0,4,1,8,2,5,7,6] 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 10 monclient(hunting): picked mon.noname-g con 0x7fe83006c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 10 monclient(hunting): picked mon.noname-a con 0x7fe8300a3860 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 10 monclient(hunting): picked mon.noname-c con 0x7fe8300a4350 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83009f7e0 con 0x7fe8300a3860 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8301ab9a0 con 0x7fe83006c1d0 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8301cb6c0 con 0x7fe8300a4350 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe83683a640 10 monclient(hunting): authenticate will time out at 748.113359s 2024-09-16T10:29:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe827fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe8300a3860 0x7fe8301cfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41608/0 (socket says 172.21.15.94:41608) 2024-09-16T10:29:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe834dd7640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fe8300a4350 0x7fe8301d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43628/0 (socket says 172.21.15.94:43628) 2024-09-16T10:29:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.773+0000 7fe827fff640 1 -- 172.21.15.94:0/3592063636 learned_addr learned my addr 172.21.15.94:0/3592063636 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.775+0000 7fe834dd7640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.94:6791/0 conn(0x7fe8300a4350 legacy=0x7fe8301d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.775+0000 7fe827fff640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.94:6789/0 conn(0x7fe8300a3860 legacy=0x7fe8301cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.775+0000 7fe82ffff640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.152:6790/0 conn(0x7fe83006c1d0 legacy=0x7fe8301be5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.776+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808403010 0 0) 0x7fe83009f7e0 con 0x7fe8300a3860 2024-09-16T10:29:40.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.776+0000 7fe82d7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.776+0000 7fe82d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.776+0000 7fe82d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.776+0000 7fe82d7fa640 10 monclient(hunting): my global_id is 4446 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.776+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe810003520 con 0x7fe8300a3860 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.776+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609111700 0 0) 0x7fe8301cb6c0 con 0x7fe8300a4350 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.776+0000 7fe82d7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): my global_id is 4468 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe83009f7e0 con 0x7fe8300a4350 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655729682 0 0) 0x7fe8301ab9a0 con 0x7fe83006c1d0 2024-09-16T10:29:40.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): my global_id is 4557 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8301cb6c0 con 0x7fe83006c1d0 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1020857551 0 0) 0x7fe83009f7e0 con 0x7fe8300a4350 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8301ab9a0 con 0x7fe8300a4350 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe828001e70 con 0x7fe8300a4350 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): dump: 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.787 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1218975826 0 0) 0x7fe8301ab9a0 con 0x7fe8300a4350 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.152:6790/0 conn(0x7fe83006c1d0 legacy=0x7fe8301be5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.777+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.94:6789/0 conn(0x7fe8300a3860 legacy=0x7fe8301cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.778+0000 7fe82d7fa640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.778+0000 7fe82d7fa640 1 monclient: found mon.c 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.778+0000 7fe82d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.778+0000 7fe82d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.778+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8301aa760 con 0x7fe8300a4350 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82d7fa640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82d7fa640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe828003c40 con 0x7fe8300a4350 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82e7fc640 4 set_mon_vals no callback set 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe8280040f0 con 0x7fe8300a4350 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe82d7fa640 10 monclient: dump: 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe83683a640 5 monclient: authenticate success, global_id 4468 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe83683a640 10 monclient: _renew_subs 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe83683a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:40.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83009f290 con 0x7fe8300a4350 2024-09-16T10:29:40.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe83683a640 10 monclient: _renew_subs 2024-09-16T10:29:40.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe83683a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:40.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.779+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8301aa4e0 con 0x7fe8300a4350 2024-09-16T10:29:40.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.780+0000 7fe83683a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.780+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe8301a9b90 con 0x7fe8301caa10 2024-09-16T10:29:40.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.781+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fe828004310 con 0x7fe8300a4350 2024-09-16T10:29:40.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.781+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3926+0+0 (unknown 1109030052 0 0) 0x7fe8280460b0 con 0x7fe8300a4350 2024-09-16T10:29:40.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.781+0000 7fe827fff640 10 monclient: get_auth_request con 0x7fe810039290 auth_method 0 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82ffff640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.152:6789/0 conn(0x7fe8301caa10 legacy=0x7fe8301a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290107988 0 0) 0x7fe8301a9b90 con 0x7fe8301caa10 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 10 monclient(hunting): my global_id is 4393 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8301ab9a0 con 0x7fe8301caa10 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1010094008 0 0) 0x7fe8301ab9a0 con 0x7fe8301caa10 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8301a9b90 con 0x7fe8301caa10 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe820004be0 con 0x7fe8301caa10 2024-09-16T10:29:40.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 10 monclient: dump: 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.794 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 469379986 0 0) 0x7fe8301a9b90 con 0x7fe8301caa10 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.784+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.152:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8300a4920 con 0x7fe8301caa10 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.785+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe820001940 con 0x7fe8301caa10 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.785+0000 7fe82d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.785+0000 7fe82d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:40.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.785+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.152:6789/0 conn(0x7fe8301caa10 legacy=0x7fe8301a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.785+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.152:6789/0 conn(0x7fe8301caa10 legacy=0x7fe8301a41c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.791+0000 7fe83683a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:29:40.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.791+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe8300665f0 con 0x7fe8301becb0 2024-09-16T10:29:40.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82ffff640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.152:6789/0 conn(0x7fe8301becb0 legacy=0x7fe8301a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003689687 0 0) 0x7fe8300665f0 con 0x7fe8301becb0 2024-09-16T10:29:40.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 10 monclient(hunting): my global_id is 4402 2024-09-16T10:29:40.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8301a9b90 con 0x7fe8301becb0 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 386331028 0 0) 0x7fe8301a9b90 con 0x7fe8301becb0 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8300665f0 con 0x7fe8301becb0 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe8200055f0 con 0x7fe8301becb0 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 10 monclient: dump: 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3842391977 0 0) 0x7fe8300665f0 con 0x7fe8301becb0 2024-09-16T10:29:40.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.794+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 --> v1:172.21.15.152:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe83004e730 con 0x7fe8301becb0 2024-09-16T10:29:40.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 <== mon.1 v1:172.21.15.152:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7452 (unknown 0 0 2706069342) 0x7fe8200078d0 con 0x7fe8301becb0 2024-09-16T10:29:40.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe82d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:29:40.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe82d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:40.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.152:6789/0 conn(0x7fe8301becb0 legacy=0x7fe8301a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe82d7fa640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.152:6789/0 conn(0x7fe8301becb0 legacy=0x7fe8301a48d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.800 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:29:40.800 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:40.800 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 8, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:29:40.801 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 24, 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.802 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.803 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.804 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.805 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.806 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.807 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.808 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.811 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.812 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:40.813 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.816 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:40.817 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.818 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:29:40.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.820 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.820 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.820 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.820 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.820 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.820 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:29:40.820 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:29:40.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.152:6800/2686360781 conn(0x7fe810039290 legacy=0x7fe81003b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe83683a640 10 monclient: shutdown 2024-09-16T10:29:40.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe83683a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 >> v1:172.21.15.94:6791/0 conn(0x7fe8300a4350 legacy=0x7fe8301d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe834dd7640 1 -- 172.21.15.94:0/3592063636 reap_dead start 2024-09-16T10:29:40.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe834dd7640 1 -- 172.21.15.94:0/3592063636 reap_dead start 2024-09-16T10:29:40.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 shutdown_connections 2024-09-16T10:29:40.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 >> 172.21.15.94:0/3592063636 conn(0x7fe8300a5960 msgr2=0x7fe8300a6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 shutdown_connections 2024-09-16T10:29:40.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.795+0000 7fe83683a640 1 -- 172.21.15.94:0/3592063636 wait complete. 2024-09-16T10:29:40.822 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:29:40.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-16T10:29:40.823 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-16T10:29:40.894 INFO:teuthology.orchestra.run.smithi152.stdout:process_1_[34831]: starting. 2024-09-16T10:29:40.895 INFO:teuthology.orchestra.run.smithi152.stdout:process_1_[34831]: creating pool ceph_test_rados_delete_pools_parallel.smithi152-34830 2024-09-16T10:29:40.895 INFO:teuthology.orchestra.run.smithi152.stdout:process_1_[34831]: created object 0... 2024-09-16T10:29:40.895 INFO:teuthology.orchestra.run.smithi152.stdout:process_1_[34831]: created object 25... 2024-09-16T10:29:40.895 INFO:teuthology.orchestra.run.smithi152.stdout:process_1_[34831]: created object 49... 2024-09-16T10:29:40.895 INFO:teuthology.orchestra.run.smithi152.stdout:process_1_[34831]: finishing. 2024-09-16T10:29:40.895 INFO:teuthology.orchestra.run.smithi152.stdout:process_1_[34831]: shutting down. 2024-09-16T10:29:40.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.925+0000 7fdd9bddb640 10 monclient: get_monmap_and_config 2024-09-16T10:29:40.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.925+0000 7fdd9bddb640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.925+0000 7fdd9bddb640 10 monclient: monmap: 2024-09-16T10:29:40.926 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.926 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.926 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:40.925828+0000 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:40.925828+0000 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.927 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.928 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.925+0000 7fdd9bddb640 1 Processor -- start 2024-09-16T10:29:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.925+0000 7fdd9bddb640 1 -- start start 2024-09-16T10:29:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.925+0000 7fdd9bddb640 10 monclient: init 2024-09-16T10:29:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 10 monclient: _add_conns ranks=[4,7,3,1,6,2,8,5,0] 2024-09-16T10:29:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 10 monclient(hunting): picked mon.noname-c con 0x7fdd940a38d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 10 monclient(hunting): picked mon.noname-i con 0x7fdd9406c670 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 10 monclient(hunting): picked mon.noname-g con 0x7fdd940a43c0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd94061ae0 con 0x7fdd940a43c0 2024-09-16T10:29:40.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd940a04d0 con 0x7fdd940a38d0 2024-09-16T10:29:40.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd940a3250 con 0x7fdd9406c670 2024-09-16T10:29:40.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd9bddb640 10 monclient(hunting): authenticate will time out at 748.266310s 2024-09-16T10:29:40.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd99b77640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fdd940a38d0 0x7fdd940a3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43630/0 (socket says 172.21.15.94:43630) 2024-09-16T10:29:40.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.926+0000 7fdd99b77640 1 -- 172.21.15.94:0/890842912 learned_addr learned my addr 172.21.15.94:0/890842912 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.928+0000 7fdd99b77640 1 -- 172.21.15.94:0/890842912 >> v1:172.21.15.94:6791/0 conn(0x7fdd940a38d0 legacy=0x7fdd940a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.928+0000 7fdd9a378640 1 -- 172.21.15.94:0/890842912 >> v1:172.21.15.152:6790/0 conn(0x7fdd940a43c0 legacy=0x7fdd9413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.928+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788998029 0 0) 0x7fdd940a04d0 con 0x7fdd940a38d0 2024-09-16T10:29:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.928+0000 7fdd98b75640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.928+0000 7fdd99376640 1 -- 172.21.15.94:0/890842912 >> v1:172.21.15.152:6792/0 conn(0x7fdd9406c670 legacy=0x7fdd9413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.928+0000 7fdd98b75640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.928+0000 7fdd98b75640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.928+0000 7fdd98b75640 10 monclient(hunting): my global_id is 4477 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.928+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd7c003520 con 0x7fdd940a38d0 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567787230 0 0) 0x7fdd940a3250 con 0x7fdd9406c670 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient(hunting): my global_id is 4462 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd940a04d0 con 0x7fdd9406c670 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151389546 0 0) 0x7fdd94061ae0 con 0x7fdd940a43c0 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient(hunting): my global_id is 4566 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd940a3250 con 0x7fdd940a43c0 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 970255844 0 0) 0x7fdd7c003520 con 0x7fdd940a38d0 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 >> v1:172.21.15.152:6792/0 conn(0x7fdd9406c670 legacy=0x7fdd9413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 >> v1:172.21.15.152:6790/0 conn(0x7fdd940a43c0 legacy=0x7fdd9413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 monclient: found mon.noname-c 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd940a0a00 con 0x7fdd940a38d0 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fdd8c001a60 con 0x7fdd940a38d0 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 5 monclient: authenticate success, global_id 4477 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: dump: 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd8c001d70 con 0x7fdd940a38d0 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 1 -- 172.21.15.94:0/890842912 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fdd8c004be0 con 0x7fdd940a38d0 2024-09-16T10:29:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 4 set_mon_vals no callback set 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd98b75640 10 monclient: dump: 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.939 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 10 monclient: shutdown 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 1 -- 172.21.15.94:0/890842912 >> v1:172.21.15.94:6791/0 conn(0x7fdd940a38d0 legacy=0x7fdd940a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 1 -- 172.21.15.94:0/890842912 shutdown_connections 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 1 -- 172.21.15.94:0/890842912 >> 172.21.15.94:0/890842912 conn(0x7fdd940a59d0 msgr2=0x7fdd940a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 1 -- 172.21.15.94:0/890842912 shutdown_connections 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.929+0000 7fdd9bddb640 1 -- 172.21.15.94:0/890842912 wait complete. 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.930+0000 7fdd9bddb640 10 monclient: build_initial_monmap 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.930+0000 7fdd9bddb640 10 monclient: monmap: 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.930+0000 7fdd9bddb640 1 Processor -- start 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.930+0000 7fdd9bddb640 1 -- start start 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.930+0000 7fdd9bddb640 10 monclient: init 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 10 monclient: _add_conns ranks=[5,2,3,0,6,8,1,7,4] 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 10 monclient(hunting): picked mon.noname-h con 0x7fdd9406c670 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 10 monclient(hunting): picked mon.noname-b con 0x7fdd940a38d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 10 monclient(hunting): picked mon.noname-g con 0x7fdd940a43c0 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd94125840 con 0x7fdd940a38d0 2024-09-16T10:29:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd94127450 con 0x7fdd940a43c0 2024-09-16T10:29:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9409f240 con 0x7fdd9406c670 2024-09-16T10:29:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd9bddb640 10 monclient(hunting): authenticate will time out at 748.271273s 2024-09-16T10:29:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd99376640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fdd940a38d0 0x7fdd941cfdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48532/0 (socket says 172.21.15.94:48532) 2024-09-16T10:29:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.931+0000 7fdd99376640 1 -- 172.21.15.94:0/2127873714 learned_addr learned my addr 172.21.15.94:0/2127873714 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:40.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd99376640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.94:6790/0 conn(0x7fdd940a38d0 legacy=0x7fdd941cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd99b77640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6791/0 conn(0x7fdd9406c670 legacy=0x7fdd94136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262395564 0 0) 0x7fdd94125840 con 0x7fdd940a38d0 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): my global_id is 4412 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd80003520 con 0x7fdd940a38d0 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd9a378640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6790/0 conn(0x7fdd940a43c0 legacy=0x7fdd941d3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 80862161 0 0) 0x7fdd80003520 con 0x7fdd940a38d0 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd94125840 con 0x7fdd940a38d0 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fdd88003610 con 0x7fdd940a38d0 2024-09-16T10:29:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): dump: 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.949 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527753559 0 0) 0x7fdd9409f240 con 0x7fdd9406c670 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 10 monclient(hunting): my global_id is 4469 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.933+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd80003520 con 0x7fdd9406c670 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689003533 0 0) 0x7fdd94127450 con 0x7fdd940a43c0 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient(hunting): my global_id is 4575 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd9409f240 con 0x7fdd940a43c0 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2071536909 0 0) 0x7fdd94125840 con 0x7fdd940a38d0 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6791/0 conn(0x7fdd9406c670 legacy=0x7fdd94136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6790/0 conn(0x7fdd940a43c0 legacy=0x7fdd941d3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient: _finish_hunting 0 2024-09-16T10:29:40.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 monclient: found mon.b 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9409eef0 con 0x7fdd940a38d0 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient: _finish_auth 0 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient: _check_auth_tickets 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd9bddb640 5 monclient: authenticate success, global_id 4412 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd9bddb640 10 monclient: _renew_subs 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd9bddb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9406ba90 con 0x7fdd940a38d0 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd9bddb640 10 monclient: _renew_subs 2024-09-16T10:29:40.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd9bddb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd94137170 con 0x7fdd940a38d0 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd88003c50 con 0x7fdd940a38d0 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fdd88004100 con 0x7fdd940a38d0 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd93fff640 4 set_mon_vals no callback set 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 10 monclient: dump: 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.953 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd9bddb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdd74006580 con 0x7fdd74002820 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.934+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fdd88004290 con 0x7fdd940a38d0 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.935+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3926+0+0 (unknown 1109030052 0 0) 0x7fdd88046220 con 0x7fdd940a38d0 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.935+0000 7fdd9a378640 10 monclient: get_auth_request con 0x7fdd800391d0 auth_method 0 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd99b77640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6790/0 conn(0x7fdd74002820 legacy=0x7fdd74004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253776754 0 0) 0x7fdd74006580 con 0x7fdd74002820 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 10 monclient(hunting): my global_id is 4584 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd94125840 con 0x7fdd74002820 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2630846678 0 0) 0x7fdd94125840 con 0x7fdd74002820 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd74006580 con 0x7fdd74002820 2024-09-16T10:29:40.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fdd8c0047e0 con 0x7fdd74002820 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 10 monclient: dump: 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 965952851 0 0) 0x7fdd74006580 con 0x7fdd74002820 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.937+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd7400a0d0 con 0x7fdd74002820 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.938+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fdd8c004cb0 con 0x7fdd74002820 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.938+0000 7fdd927fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.938+0000 7fdd927fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:40.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.938+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6790/0 conn(0x7fdd74002820 legacy=0x7fdd74004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.938+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6790/0 conn(0x7fdd74002820 legacy=0x7fdd74004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.944+0000 7fdd9bddb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:29:40.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.944+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdd74009d90 con 0x7fdd740090b0 2024-09-16T10:29:40.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.946+0000 7fdd99b77640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6790/0 conn(0x7fdd740090b0 legacy=0x7fdd74009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:40.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688968691 0 0) 0x7fdd74009d90 con 0x7fdd740090b0 2024-09-16T10:29:40.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 10 monclient(hunting): _negotiate 2024-09-16T10:29:40.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:40.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 10 monclient(hunting): my global_id is 4593 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd74006580 con 0x7fdd740090b0 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1029406154 0 0) 0x7fdd74006580 con 0x7fdd740090b0 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd74009d90 con 0x7fdd740090b0 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fdd8c005530 con 0x7fdd740090b0 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 10 monclient: dump: 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:40.960 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:40.961 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2933971902 0 0) 0x7fdd74009d90 con 0x7fdd740090b0 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 --> v1:172.21.15.152:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdd7400c280 con 0x7fdd740090b0 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 <== mon.3 v1:172.21.15.152:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7618 (unknown 0 0 2134141751) 0x7fdd8c0077b0 con 0x7fdd740090b0 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6790/0 conn(0x7fdd740090b0 legacy=0x7fdd74009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.947+0000 7fdd927fc640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6790/0 conn(0x7fdd740090b0 legacy=0x7fdd74009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:40.962 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 8, 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:29:40.963 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 24, 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.964 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.965 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:29:40.966 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:29:40.967 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:40.968 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:29:40.969 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.970 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.972 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.975 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.976 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:29:40.976 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.976 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.976 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.976 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:40.976 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.978 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:29:40.979 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:40.980 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:29:40.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:29:40.983 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:29:40.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.152:6800/2686360781 conn(0x7fdd800391d0 legacy=0x7fdd8003b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9bddb640 10 monclient: shutdown 2024-09-16T10:29:40.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9bddb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:40.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 >> v1:172.21.15.94:6790/0 conn(0x7fdd940a38d0 legacy=0x7fdd941cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:40.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9a378640 1 -- 172.21.15.94:0/2127873714 reap_dead start 2024-09-16T10:29:40.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9a378640 1 -- 172.21.15.94:0/2127873714 reap_dead start 2024-09-16T10:29:40.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 shutdown_connections 2024-09-16T10:29:40.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 >> 172.21.15.94:0/2127873714 conn(0x7fdd940a59d0 msgr2=0x7fdd940a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:40.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 shutdown_connections 2024-09-16T10:29:40.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:40.948+0000 7fdd9bddb640 1 -- 172.21.15.94:0/2127873714 wait complete. 2024-09-16T10:29:40.986 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:29:40.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-16T10:29:40.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-16T10:29:41.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.084+0000 7f1db0bae640 10 monclient: get_monmap_and_config 2024-09-16T10:29:41.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.084+0000 7f1db0bae640 10 monclient: build_initial_monmap 2024-09-16T10:29:41.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.084+0000 7f1db0bae640 10 monclient: monmap: 2024-09-16T10:29:41.086 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:41.085502+0000 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:41.085502+0000 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:41.087 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:41.088 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:41.088 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:41.088 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:41.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 1 Processor -- start 2024-09-16T10:29:41.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 1 -- start start 2024-09-16T10:29:41.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 10 monclient: init 2024-09-16T10:29:41.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 10 monclient: _add_conns ranks=[7,0,3,5,1,2,6,8,4] 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 10 monclient(hunting): picked mon.noname-i con 0x7f1dac0a3860 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 10 monclient(hunting): picked mon.noname-a con 0x7f1dac06c1d0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 10 monclient(hunting): picked mon.noname-g con 0x7f1dac0a4350 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac061ae0 con 0x7f1dac06c1d0 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0a0600 con 0x7f1dac0a4350 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0a0b30 con 0x7f1dac0a3860 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1db0bae640 10 monclient(hunting): authenticate will time out at 748.425801s 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1da1d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1dac06c1d0 0x7f1dac13b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41622/0 (socket says 172.21.15.94:41622) 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.085+0000 7f1da1d9b640 1 -- 172.21.15.94:0/3990535891 learned_addr learned my addr 172.21.15.94:0/3990535891 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:41.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da1d9b640 1 -- 172.21.15.94:0/3990535891 >> v1:172.21.15.94:6789/0 conn(0x7f1dac06c1d0 legacy=0x7f1dac13b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 -- 172.21.15.94:0/3990535891 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510450242 0 0) 0x7f1dac061ae0 con 0x7f1dac06c1d0 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient(hunting): _negotiate 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient(hunting): my global_id is 4455 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 -- 172.21.15.94:0/3990535891 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d98003520 con 0x7f1dac06c1d0 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1daa59c640 1 -- 172.21.15.94:0/3990535891 >> v1:172.21.15.152:6792/0 conn(0x7f1dac0a3860 legacy=0x7f1dac0a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1daad9d640 1 -- 172.21.15.94:0/3990535891 >> v1:172.21.15.152:6790/0 conn(0x7f1dac0a4350 legacy=0x7f1dac13ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 -- 172.21.15.94:0/3990535891 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746686619 0 0) 0x7f1d98003520 con 0x7f1dac06c1d0 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 -- 172.21.15.94:0/3990535891 >> v1:172.21.15.152:6792/0 conn(0x7f1dac0a3860 legacy=0x7f1dac0a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 -- 172.21.15.94:0/3990535891 >> v1:172.21.15.152:6790/0 conn(0x7f1dac0a4350 legacy=0x7f1dac13ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:41.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: _finish_hunting 0 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 monclient: found mon.noname-a 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 -- 172.21.15.94:0/3990535891 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac0a0e10 con 0x7f1dac06c1d0 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: _finish_auth 0 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: _check_auth_tickets 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 -- 172.21.15.94:0/3990535891 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1d94001a50 con 0x7f1dac06c1d0 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1db0bae640 5 monclient: authenticate success, global_id 4455 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1db0bae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:29:41.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: dump: 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1db0bae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 -- 172.21.15.94:0/3990535891 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d94001d60 con 0x7f1dac06c1d0 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 1 -- 172.21.15.94:0/3990535891 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1d94004be0 con 0x7f1dac06c1d0 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1db0bae640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1db0bae640 4 set_mon_vals no callback set 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:29:41.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1da9d9b640 10 monclient: dump: 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:41.097 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1db0bae640 10 monclient: shutdown 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1db0bae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.088+0000 7f1db0bae640 1 -- 172.21.15.94:0/3990535891 >> v1:172.21.15.94:6789/0 conn(0x7f1dac06c1d0 legacy=0x7f1dac13b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.089+0000 7f1db0bae640 1 -- 172.21.15.94:0/3990535891 shutdown_connections 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.089+0000 7f1db0bae640 1 -- 172.21.15.94:0/3990535891 >> 172.21.15.94:0/3990535891 conn(0x7f1dac0a5960 msgr2=0x7f1dac0a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.089+0000 7f1db0bae640 1 -- 172.21.15.94:0/3990535891 shutdown_connections 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.089+0000 7f1db0bae640 1 -- 172.21.15.94:0/3990535891 wait complete. 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.089+0000 7f1db0bae640 10 monclient: build_initial_monmap 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.089+0000 7f1db0bae640 10 monclient: monmap: 2024-09-16T10:29:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:41.099 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:41.100 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:41.100 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:41.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 1 Processor -- start 2024-09-16T10:29:41.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 1 -- start start 2024-09-16T10:29:41.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 10 monclient: init 2024-09-16T10:29:41.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:41.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 10 monclient: _add_conns ranks=[2,7,4,1,8,3,6,0,5] 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 10 monclient(hunting): picked mon.noname-b con 0x7f1dac06c1d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 10 monclient(hunting): picked mon.noname-i con 0x7f1dac0a3860 addr v1:172.21.15.152:6792/0 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 10 monclient(hunting): picked mon.noname-c con 0x7f1dac0a4350 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac06bae0 con 0x7f1dac06c1d0 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac1a8f40 con 0x7f1dac0a4350 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac1a6be0 con 0x7f1dac0a3860 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1db0bae640 10 monclient(hunting): authenticate will time out at 748.430795s 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1daa59c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f1dac06c1d0 0x7f1dac1bb560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48544/0 (socket says 172.21.15.94:48544) 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.090+0000 7f1daa59c640 1 -- 172.21.15.94:0/2970148201 learned_addr learned my addr 172.21.15.94:0/2970148201 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:41.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.092+0000 7f1daa59c640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.94:6790/0 conn(0x7f1dac06c1d0 legacy=0x7f1dac1bb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:41.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.092+0000 7f1daad9d640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.94:6791/0 conn(0x7f1dac0a4350 legacy=0x7f1dac1d34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:41.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.093+0000 7f1da1d9b640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.152:6792/0 conn(0x7f1dac0a3860 legacy=0x7f1dac1cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:41.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.876+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342389932 0 0) 0x7f1dac06bae0 con 0x7f1dac06c1d0 2024-09-16T10:29:41.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.876+0000 7f1da37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:41.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.876+0000 7f1da37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.876+0000 7f1da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.876+0000 7f1da37fe640 10 monclient(hunting): my global_id is 4421 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.876+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c003520 con 0x7f1dac06c1d0 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61736860 0 0) 0x7f1dac1a6be0 con 0x7f1dac0a3860 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 10 monclient(hunting): my global_id is 4471 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dac06bae0 con 0x7f1dac0a3860 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595087426 0 0) 0x7f1dac1a8f40 con 0x7f1dac0a4350 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:41.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:41.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 10 monclient(hunting): my global_id is 4486 2024-09-16T10:29:41.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.877+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dac1a6be0 con 0x7f1dac0a4350 2024-09-16T10:29:41.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.997+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 169466795 0 0) 0x7f1dac06bae0 con 0x7f1dac0a3860 2024-09-16T10:29:41.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.997+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dac1a8f40 con 0x7f1dac0a3860 2024-09-16T10:29:41.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.997+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1d94004b20 con 0x7f1dac0a3860 2024-09-16T10:29:41.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.997+0000 7f1da37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:41.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.997+0000 7f1da37fe640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:29:41.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.997+0000 7f1da37fe640 10 monclient(hunting): dump: 2024-09-16T10:29:41.999 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.001 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.001 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.997+0000 7f1da37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.997+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2159013879 0 0) 0x7f1dac1a8f40 con 0x7f1dac0a3860 2024-09-16T10:29:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.998+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.94:6791/0 conn(0x7f1dac0a4350 legacy=0x7f1dac1d34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.998+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.94:6790/0 conn(0x7f1dac06c1d0 legacy=0x7f1dac1bb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.998+0000 7f1da37fe640 10 monclient: _finish_hunting 0 2024-09-16T10:29:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.998+0000 7f1da37fe640 1 monclient: found mon.i 2024-09-16T10:29:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.998+0000 7f1da37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.999+0000 7f1da37fe640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.999+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac09f870 con 0x7f1dac0a3860 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.999+0000 7f1da37fe640 10 monclient: _finish_auth 0 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.999+0000 7f1da37fe640 10 monclient: _check_auth_tickets 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.999+0000 7f1da37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.999+0000 7f1db0bae640 5 monclient: authenticate success, global_id 4471 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.999+0000 7f1db0bae640 10 monclient: _renew_subs 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.999+0000 7f1db0bae640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:41.999+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac09f320 con 0x7f1dac0a3860 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.000+0000 7f1db0bae640 10 monclient: _renew_subs 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.001+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d94001a30 con 0x7f1dac0a3860 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.001+0000 7f1db0bae640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:29:42.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.001+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac1a7500 con 0x7f1dac0a3860 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.001+0000 7f1da37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.001+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1d94004b20 con 0x7f1dac0a3860 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.001+0000 7f1da37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.001+0000 7f1da37fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.001+0000 7f1da37fe640 10 monclient: dump: 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.005 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.001+0000 7f1da959a640 4 set_mon_vals no callback set 2024-09-16T10:29:42.007 INFO:teuthology.orchestra.run.smithi152.stdout:process_2_[34832]: starting. 2024-09-16T10:29:42.007 INFO:teuthology.orchestra.run.smithi152.stdout:process_2_[34832]: deleting pool ceph_test_rados_delete_pools_parallel.smithi152-34830 2024-09-16T10:29:42.007 INFO:teuthology.orchestra.run.smithi152.stdout:process_2_[34832]: shutting down. 2024-09-16T10:29:42.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.002+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f1d940100d0 con 0x7f1dac0a3860 2024-09-16T10:29:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.003+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3506+0+0 (unknown 4039528226 0 0) 0x7f1d94014d80 con 0x7f1dac0a3860 2024-09-16T10:29:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.003+0000 7f1db0bae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.003+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1d90006580 con 0x7f1d90002820 2024-09-16T10:29:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.004+0000 7f1daa59c640 10 monclient: get_auth_request con 0x7f1d8c0394b0 auth_method 0 2024-09-16T10:29:42.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.006+0000 7f1daad9d640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.152:6791/0 conn(0x7f1d90002820 legacy=0x7f1d90004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022968994 0 0) 0x7f1d90006580 con 0x7f1d90002820 2024-09-16T10:29:42.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 10 monclient(hunting): my global_id is 4478 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c004340 con 0x7f1d90002820 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1407493148 0 0) 0x7f1d8c004340 con 0x7f1d90002820 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d90006580 con 0x7f1d90002820 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1da4001cb0 con 0x7f1d90002820 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 10 monclient: dump: 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.010 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2243405779 0 0) 0x7f1d90006580 con 0x7f1d90002820 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.007+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d9000a0d0 con 0x7f1d90002820 2024-09-16T10:29:42.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.008+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1da4005b40 con 0x7f1d90002820 2024-09-16T10:29:42.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.008+0000 7f1da37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:42.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.008+0000 7f1da37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:42.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.008+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.152:6791/0 conn(0x7f1d90002820 legacy=0x7f1d90004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.008+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.152:6791/0 conn(0x7f1d90002820 legacy=0x7f1d90004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:42.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.015+0000 7f1db0bae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:29:42.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.015+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1d90009d70 con 0x7f1d90009090 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1daad9d640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.152:6791/0 conn(0x7f1d90009090 legacy=0x7f1d90009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585466997 0 0) 0x7f1d90009d70 con 0x7f1d90009090 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 10 monclient(hunting): my global_id is 4496 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d90006580 con 0x7f1d90009090 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2797045938 0 0) 0x7f1d90006580 con 0x7f1d90009090 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d90009d70 con 0x7f1d90009090 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1da4004080 con 0x7f1d90009090 2024-09-16T10:29:42.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.017+0000 7f1da37fe640 10 monclient: dump: 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1249806482 0 0) 0x7f1d90009d70 con 0x7f1d90009090 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 --> v1:172.21.15.152:6791/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1d9000c280 con 0x7f1d90009090 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 <== mon.5 v1:172.21.15.152:6791/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7618 (unknown 0 0 277328034) 0x7f1da4004550 con 0x7f1d90009090 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1da37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1da37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.152:6791/0 conn(0x7f1d90009090 legacy=0x7f1d90009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1da37fe640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.152:6791/0 conn(0x7f1d90009090 legacy=0x7f1d90009520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:42.023 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:29:42.023 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:42.023 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:42.023 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:29:42.023 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 8, 2024-09-16T10:29:42.023 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:29:42.023 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:29:42.023 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:29:42.023 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 26, 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:29:42.024 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:42.025 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:29:42.026 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:29:42.027 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:42.028 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.029 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.030 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:29:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:29:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:29:42.043 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:29:42.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.152:6800/2686360781 conn(0x7f1d8c0394b0 legacy=0x7f1d8c03b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1db0bae640 10 monclient: shutdown 2024-09-16T10:29:42.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1db0bae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:42.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 >> v1:172.21.15.152:6792/0 conn(0x7f1dac0a3860 legacy=0x7f1dac1cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1daad9d640 1 -- 172.21.15.94:0/2970148201 reap_dead start 2024-09-16T10:29:42.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1daad9d640 1 -- 172.21.15.94:0/2970148201 reap_dead start 2024-09-16T10:29:42.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 shutdown_connections 2024-09-16T10:29:42.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 >> 172.21.15.94:0/2970148201 conn(0x7f1dac0a5960 msgr2=0x7f1dac0a7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:42.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.018+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 shutdown_connections 2024-09-16T10:29:42.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.019+0000 7f1db0bae640 1 -- 172.21.15.94:0/2970148201 wait complete. 2024-09-16T10:29:42.045 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:29:42.045 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-16T10:29:42.046 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-16T10:29:42.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.141+0000 7fc38e365640 10 monclient: get_monmap_and_config 2024-09-16T10:29:42.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.141+0000 7fc38e365640 10 monclient: build_initial_monmap 2024-09-16T10:29:42.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.142+0000 7fc38e365640 10 monclient: monmap: 2024-09-16T10:29:42.143 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:42.143 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:42.142792+0000 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:42.142792+0000 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:42.144 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:42.145 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.142+0000 7fc38e365640 1 Processor -- start 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.142+0000 7fc38e365640 1 -- start start 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.142+0000 7fc38e365640 10 monclient: init 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 10 monclient: _add_conns ranks=[1,6,5,7,4,3,8,2,0] 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 10 monclient(hunting): picked mon.noname-f con 0x7fc3880a3840 addr v1:172.21.15.152:6789/0 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 10 monclient(hunting): picked mon.noname-d con 0x7fc3880a17d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 10 monclient(hunting): picked mon.noname-h con 0x7fc3880a4330 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc388061ae0 con 0x7fc3880a3840 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3880a3280 con 0x7fc3880a4330 2024-09-16T10:29:42.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38806c7d0 con 0x7fc3880a17d0 2024-09-16T10:29:42.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:42.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc38e365640 10 monclient(hunting): authenticate will time out at 749.483443s 2024-09-16T10:29:42.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc3877fe640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fc3880a17d0 0x7fc38813ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:60418/0 (socket says 172.21.15.94:60418) 2024-09-16T10:29:42.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.143+0000 7fc3877fe640 1 -- 172.21.15.94:0/1028742246 learned_addr learned my addr 172.21.15.94:0/1028742246 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:42.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.145+0000 7fc3877fe640 1 -- 172.21.15.94:0/1028742246 >> v1:172.21.15.94:6792/0 conn(0x7fc3880a17d0 legacy=0x7fc38813ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.145+0000 7fc387fff640 1 -- 172.21.15.94:0/1028742246 >> v1:172.21.15.152:6789/0 conn(0x7fc3880a3840 legacy=0x7fc3880a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.145+0000 7fc38c902640 1 -- 172.21.15.94:0/1028742246 >> v1:172.21.15.152:6791/0 conn(0x7fc3880a4330 legacy=0x7fc38813f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918865934 0 0) 0x7fc38806c7d0 con 0x7fc3880a17d0 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): my global_id is 4452 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc370003520 con 0x7fc3880a17d0 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346014461 0 0) 0x7fc388061ae0 con 0x7fc3880a3840 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): my global_id is 4420 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc38806c7d0 con 0x7fc3880a3840 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264921509 0 0) 0x7fc3880a3280 con 0x7fc3880a4330 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient(hunting): my global_id is 4505 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc388061ae0 con 0x7fc3880a4330 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4263459574 0 0) 0x7fc370003520 con 0x7fc3880a17d0 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 >> v1:172.21.15.152:6791/0 conn(0x7fc3880a4330 legacy=0x7fc38813f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 >> v1:172.21.15.152:6789/0 conn(0x7fc3880a3840 legacy=0x7fc3880a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 monclient: found mon.noname-d 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3880a0550 con 0x7fc3880a17d0 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: _finish_auth 0 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc37c001a60 con 0x7fc3880a17d0 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: dump: 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.153 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.154 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 5 monclient: authenticate success, global_id 4452 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc37c001d70 con 0x7fc3880a17d0 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 1 -- 172.21.15.94:0/1028742246 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc37c004be0 con 0x7fc3880a17d0 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 10 monclient: get_monmap_and_config success 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 4 set_mon_vals no callback set 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc386ffd640 10 monclient: dump: 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 10 monclient: shutdown 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 1 -- 172.21.15.94:0/1028742246 >> v1:172.21.15.94:6792/0 conn(0x7fc3880a17d0 legacy=0x7fc38813ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 1 -- 172.21.15.94:0/1028742246 shutdown_connections 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 1 -- 172.21.15.94:0/1028742246 >> 172.21.15.94:0/1028742246 conn(0x7fc3880a97b0 msgr2=0x7fc3880a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.146+0000 7fc38e365640 1 -- 172.21.15.94:0/1028742246 shutdown_connections 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.147+0000 7fc38e365640 1 -- 172.21.15.94:0/1028742246 wait complete. 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.147+0000 7fc38e365640 10 monclient: build_initial_monmap 2024-09-16T10:29:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.147+0000 7fc38e365640 10 monclient: monmap: 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:42.158 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.147+0000 7fc38e365640 1 Processor -- start 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 1 -- start start 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 10 monclient: init 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 10 monclient: _add_conns ranks=[4,5,3,8,1,7,2,6,0] 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 10 monclient(hunting): picked mon.noname-c con 0x7fc3880a3840 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 10 monclient(hunting): picked mon.noname-h con 0x7fc3880a4330 addr v1:172.21.15.152:6791/0 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 10 monclient(hunting): picked mon.noname-g con 0x7fc388137210 addr v1:172.21.15.152:6790/0 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38806ba10 con 0x7fc388137210 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3881274f0 con 0x7fc3880a3840 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3881256a0 con 0x7fc3880a4330 2024-09-16T10:29:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:42.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc38e365640 10 monclient(hunting): authenticate will time out at 749.488501s 2024-09-16T10:29:42.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc387fff640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fc3880a3840 0x7fc3881caba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:43650/0 (socket says 172.21.15.94:43650) 2024-09-16T10:29:42.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.148+0000 7fc387fff640 1 -- 172.21.15.94:0/3328965983 learned_addr learned my addr 172.21.15.94:0/3328965983 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.150+0000 7fc387fff640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.94:6791/0 conn(0x7fc3880a3840 legacy=0x7fc3881caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.150+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250758783 0 0) 0x7fc3881274f0 con 0x7fc3880a3840 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.150+0000 7fc384ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.150+0000 7fc384ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.150+0000 7fc384ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.150+0000 7fc384ff9640 10 monclient(hunting): my global_id is 4513 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.150+0000 7fc3877fe640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6791/0 conn(0x7fc3880a4330 legacy=0x7fc388136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.150+0000 7fc38c902640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6790/0 conn(0x7fc388137210 legacy=0x7fc3881d2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.150+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc374003520 con 0x7fc3880a3840 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292359325 0 0) 0x7fc3881256a0 con 0x7fc3880a4330 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): my global_id is 4514 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3881274f0 con 0x7fc3880a4330 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859370271 0 0) 0x7fc38806ba10 con 0x7fc388137210 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): my global_id is 4620 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3881256a0 con 0x7fc388137210 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3576024592 0 0) 0x7fc374003520 con 0x7fc3880a3840 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc38806ba10 con 0x7fc3880a3840 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc378003610 con 0x7fc3880a3840 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): dump: 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.166 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3729975140 0 0) 0x7fc38806ba10 con 0x7fc3880a3840 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6791/0 conn(0x7fc3880a4330 legacy=0x7fc388136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6790/0 conn(0x7fc388137210 legacy=0x7fc3881d2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient: _finish_hunting 0 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 monclient: found mon.c 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38809f360 con 0x7fc3880a3840 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient: _finish_auth 0 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient: _check_auth_tickets 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc38e365640 5 monclient: authenticate success, global_id 4513 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc38e365640 10 monclient: _renew_subs 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc38e365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38809f030 con 0x7fc3880a3840 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc38e365640 10 monclient: _renew_subs 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc38e365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc388125190 con 0x7fc3880a3840 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc378003c30 con 0x7fc3880a3840 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:29:42.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc3780040e0 con 0x7fc3880a3840 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc3867fc640 4 set_mon_vals no callback set 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.151+0000 7fc384ff9640 10 monclient: dump: 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.169 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.152+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fc3780042c0 con 0x7fc3880a3840 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.152+0000 7fc38e365640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.152+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc368006580 con 0x7fc368002820 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.152+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3506+0+0 (unknown 4039528226 0 0) 0x7fc378046090 con 0x7fc3880a3840 2024-09-16T10:29:42.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.152+0000 7fc3877fe640 10 monclient: get_auth_request con 0x7fc3740391f0 auth_method 0 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.154+0000 7fc38c902640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6792/0 conn(0x7fc368002820 legacy=0x7fc368004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.154+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872902428 0 0) 0x7fc368006580 con 0x7fc368002820 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.154+0000 7fc384ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.154+0000 7fc384ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.154+0000 7fc384ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.154+0000 7fc384ff9640 10 monclient(hunting): my global_id is 4498 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.154+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc38806ba10 con 0x7fc368002820 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.154+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3419079459 0 0) 0x7fc38806ba10 con 0x7fc368002820 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.154+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc368006580 con 0x7fc368002820 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc380003f40 con 0x7fc368002820 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:42.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 10 monclient: dump: 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 753822313 0 0) 0x7fc368006580 con 0x7fc368002820 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc36800a0d0 con 0x7fc368002820 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc380005b40 con 0x7fc368002820 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6792/0 conn(0x7fc368002820 legacy=0x7fc368004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.155+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6792/0 conn(0x7fc368002820 legacy=0x7fc368004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:42.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.161+0000 7fc38e365640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:29:42.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.161+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc368009d70 con 0x7fc368009090 2024-09-16T10:29:42.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc38c902640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6792/0 conn(0x7fc368009090 legacy=0x7fc368009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:42.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579624566 0 0) 0x7fc368009d70 con 0x7fc368009090 2024-09-16T10:29:42.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:29:42.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:29:42.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:29:42.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 10 monclient(hunting): my global_id is 4507 2024-09-16T10:29:42.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc368006580 con 0x7fc368009090 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1498983483 0 0) 0x7fc368006580 con 0x7fc368009090 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc368009d70 con 0x7fc368009090 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc380005d10 con 0x7fc368009090 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 10 monclient: dump: 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:29:42.176 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 995182679 0 0) 0x7fc368009d70 con 0x7fc368009090 2024-09-16T10:29:42.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.164+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 --> v1:172.21.15.152:6792/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc36800c280 con 0x7fc368009090 2024-09-16T10:29:42.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 <== mon.7 v1:172.21.15.152:6792/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7452 (unknown 0 0 2494453061) 0x7fc380006100 con 0x7fc368009090 2024-09-16T10:29:42.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc384ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:29:42.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc384ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:29:42.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6792/0 conn(0x7fc368009090 legacy=0x7fc368009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc384ff9640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6792/0 conn(0x7fc368009090 legacy=0x7fc368009520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:29:42.178 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:29:42.178 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:42.178 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:42.178 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 8, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 26, 2024-09-16T10:29:42.179 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:42.180 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:42.181 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:29:42.182 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:29:42.183 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:29:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.185 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:29:42.186 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.187 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.189 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.190 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:29:42.190 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:29:42.190 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.190 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.190 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.190 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:29:42.191 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:29:42.192 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:29:42.193 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.195 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:29:42.196 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:29:42.197 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 3 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:29:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:29:42.199 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:29:42.199 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:29:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.152:6800/2686360781 conn(0x7fc3740391f0 legacy=0x7fc37403b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38e365640 10 monclient: shutdown 2024-09-16T10:29:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38e365640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:29:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 >> v1:172.21.15.94:6791/0 conn(0x7fc3880a3840 legacy=0x7fc3881caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:29:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38c902640 1 -- 172.21.15.94:0/3328965983 reap_dead start 2024-09-16T10:29:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38c902640 1 -- 172.21.15.94:0/3328965983 reap_dead start 2024-09-16T10:29:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 shutdown_connections 2024-09-16T10:29:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 >> 172.21.15.94:0/3328965983 conn(0x7fc3880a97b0 msgr2=0x7fc3880a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:29:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 shutdown_connections 2024-09-16T10:29:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:42.165+0000 7fc38e365640 1 -- 172.21.15.94:0/3328965983 wait complete. 2024-09-16T10:29:42.201 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:29:42.201 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['d'] 2024-09-16T10:29:42.201 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['f'] 2024-09-16T10:29:42.201 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-09-16T10:29:42.201 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-09-16T10:29:42.202 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-16T10:29:42.202 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:29:42.212 INFO:tasks.ceph.mon.d:Stopped 2024-09-16T10:29:42.212 INFO:tasks.mon_thrash:Sending STOP to mon f 2024-09-16T10:29:42.212 INFO:tasks.ceph.mon.f:Sent signal 19 2024-09-16T10:29:42.212 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T10:29:57.213 INFO:tasks.mon_thrash:Sending CONT to mon f 2024-09-16T10:29:57.214 INFO:tasks.ceph.mon.f:Sent signal 18 2024-09-16T10:29:57.214 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-09-16T10:29:57.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.319+0000 7f8b32b65640 10 monclient: get_monmap_and_config 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.319+0000 7f8b32b65640 10 monclient: build_initial_monmap 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.319+0000 7f8b32b65640 10 monclient: monmap: 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:29:57.320246+0000 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:29:57.320246+0000 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:29:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:29:57.322 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:29:57.322 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:29:57.322 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:29:57.322 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:29:57.322 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:29:57.322 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:29:57.322 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:29:57.322 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:29:57.322 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:29:57.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 1 Processor -- start 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 1 -- start start 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 10 monclient: init 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 10 monclient: _reopen_session rank -1 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 10 monclient: _add_conns ranks=[8,4,0,2,5,1,7,3,6] 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 10 monclient(hunting): picked mon.noname-e con 0x7f8b2c09bb70 addr v1:172.21.15.94:6793/0 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 10 monclient(hunting): picked mon.noname-c con 0x7f8b2c09ede0 addr v1:172.21.15.94:6791/0 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 10 monclient(hunting): picked mon.noname-a con 0x7f8b2c0a0aa0 addr v1:172.21.15.94:6789/0 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2c061ae0 con 0x7f8b2c0a0aa0 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2c09dbe0 con 0x7f8b2c09ede0 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2c09d900 con 0x7f8b2c09bb70 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 10 monclient(hunting): _renew_subs 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b32b65640 10 monclient(hunting): authenticate will time out at 764.660758s 2024-09-16T10:29:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b30901640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f8b2c09bb70 0x7f8b2c09bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:52040/0 (socket says 172.21.15.94:52040) 2024-09-16T10:29:57.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b2bfff640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f8b2c09ede0 0x7f8b2c138590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:45754/0 (socket says 172.21.15.94:45754) 2024-09-16T10:29:57.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b30901640 1 -- 172.21.15.94:0/1843700520 learned_addr learned my addr 172.21.15.94:0/1843700520 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:29:57.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.320+0000 7f8b31102640 1 --1- 172.21.15.94:0/1843700520 >> v1:172.21.15.94:6789/0 conn(0x7f8b2c0a0aa0 0x7f8b2c13bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42070/0 (socket says 172.21.15.94:42070) 2024-09-16T10:29:57.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.323+0000 7f8b30901640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.94:6793/0 conn(0x7f8b2c09bb70 legacy=0x7f8b2c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:57.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.323+0000 7f8b2bfff640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.94:6791/0 conn(0x7f8b2c09ede0 legacy=0x7f8b2c138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:29:57.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:29:57.323+0000 7f8b31102640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.94:6789/0 conn(0x7f8b2c0a0aa0 legacy=0x7f8b2c13bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:00.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 10 monclient(hunting): tick 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 1 monclient(hunting): continuing hunt 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.94:6793/0 conn(0x7f8b2c09bb70 legacy=0x7f8b2c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.94:6791/0 conn(0x7f8b2c09ede0 legacy=0x7f8b2c138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.94:6789/0 conn(0x7f8b2c0a0aa0 legacy=0x7f8b2c13bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 10 monclient: _add_conns ranks=[1,2,3,6,5,7] 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 10 monclient(hunting): picked mon.noname-f con 0x7f8b10005b70 addr v1:172.21.15.152:6789/0 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f8b10009760 addr v1:172.21.15.94:6790/0 2024-09-16T10:30:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 10 monclient(hunting): picked mon.noname-g con 0x7f8b1000d4b0 addr v1:172.21.15.152:6790/0 2024-09-16T10:30:00.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 1 -- 172.21.15.94:0/1843700520 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2c09d620 con 0x7f8b10005b70 2024-09-16T10:30:00.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 1 -- 172.21.15.94:0/1843700520 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b10012350 con 0x7f8b10009760 2024-09-16T10:30:00.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 1 -- 172.21.15.94:0/1843700520 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100135a0 con 0x7f8b1000d4b0 2024-09-16T10:30:00.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.321+0000 7f8b2a7fc640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:00.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.323+0000 7f8b2bfff640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.94:6790/0 conn(0x7f8b10009760 legacy=0x7f8b1000bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:00.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.323+0000 7f8b30901640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.152:6789/0 conn(0x7f8b10005b70 legacy=0x7f8b10007fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:00.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:00.323+0000 7f8b31102640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.152:6790/0 conn(0x7f8b1000d4b0 legacy=0x7f8b1000f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307742595 0 0) 0x7f8b100135a0 con 0x7f8b1000d4b0 2024-09-16T10:30:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 10 monclient(hunting): my global_id is 4629 2024-09-16T10:30:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1c003520 con 0x7f8b1000d4b0 2024-09-16T10:30:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833626918 0 0) 0x7f8b10012350 con 0x7f8b10009760 2024-09-16T10:30:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 10 monclient(hunting): my global_id is 4448 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.221+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b100135a0 con 0x7f8b10009760 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4170327273 0 0) 0x7f8b1c003520 con 0x7f8b1000d4b0 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.94:6790/0 conn(0x7f8b10009760 legacy=0x7f8b1000bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.152:6789/0 conn(0x7f8b10005b70 legacy=0x7f8b10007fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 1 monclient: found mon.noname-g 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.152:6790/0 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b100147f0 con 0x7f8b1000d4b0 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: _finish_auth 0 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:30:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8b24001da0 con 0x7f8b1000d4b0 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b31102640 1 -- 172.21.15.94:0/1843700520 reap_dead start 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: dump: 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.227 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b32b65640 5 monclient: authenticate success, global_id 4629 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b32b65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 <== mon.3 v1:172.21.15.152:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b240020b0 con 0x7f8b1000d4b0 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1843700520 <== mon.3 v1:172.21.15.152:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8b24004ea0 con 0x7f8b1000d4b0 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b2b7fe640 10 monclient: dump: 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.229 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b32b65640 10 monclient: get_monmap_and_config success 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.222+0000 7f8b32b65640 4 set_mon_vals no callback set 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.223+0000 7f8b32b65640 10 monclient: shutdown 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.223+0000 7f8b32b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.223+0000 7f8b32b65640 1 -- 172.21.15.94:0/1843700520 >> v1:172.21.15.152:6790/0 conn(0x7f8b1000d4b0 legacy=0x7f8b1000f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.223+0000 7f8b32b65640 1 -- 172.21.15.94:0/1843700520 shutdown_connections 2024-09-16T10:30:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.223+0000 7f8b32b65640 1 -- 172.21.15.94:0/1843700520 >> 172.21.15.94:0/1843700520 conn(0x7f8b2c0a1680 msgr2=0x7f8b2c0a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:02.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.223+0000 7f8b32b65640 1 -- 172.21.15.94:0/1843700520 shutdown_connections 2024-09-16T10:30:02.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.223+0000 7f8b32b65640 1 -- 172.21.15.94:0/1843700520 wait complete. 2024-09-16T10:30:02.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.224+0000 7f8b32b65640 10 monclient: build_initial_monmap 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.224+0000 7f8b32b65640 10 monclient: monmap: 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.224+0000 7f8b32b65640 1 Processor -- start 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.224+0000 7f8b32b65640 1 -- start start 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.224+0000 7f8b32b65640 10 monclient: init 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 10 monclient: _add_conns ranks=[6,3,0,2,5,1,7,4,8] 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 10 monclient(hunting): picked mon.noname-d con 0x7f8b1000d4b0 addr v1:172.21.15.94:6792/0 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 10 monclient(hunting): picked mon.noname-g con 0x7f8b2c1ca8d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 10 monclient(hunting): picked mon.noname-a con 0x7f8b2c1d1330 addr v1:172.21.15.94:6789/0 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2c070610 con 0x7f8b2c1d1330 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2c0a3410 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2c1cacb0 con 0x7f8b1000d4b0 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b32b65640 10 monclient(hunting): authenticate will time out at 769.565576s 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b30901640 1 -- >> v1:172.21.15.94:6792/0 conn(0x7f8b1000d4b0 legacy=0x7f8b2c133b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6792/0 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b2bfff640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7f8b2c1ca8d0 0x7f8b2c1cfc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.94:49326/0 (socket says 172.21.15.94:49326) 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b31102640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8b2c1d1330 0x7f8b2c1d3720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42084/0 (socket says 172.21.15.94:42084) 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.225+0000 7f8b2bfff640 1 -- 172.21.15.94:0/4009944231 learned_addr learned my addr 172.21.15.94:0/4009944231 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b31102640 1 -- 172.21.15.94:0/4009944231 >> v1:172.21.15.94:6789/0 conn(0x7f8b2c1d1330 legacy=0x7f8b2c1d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b2bfff640 1 -- 172.21.15.94:0/4009944231 >> v1:172.21.15.152:6790/0 conn(0x7f8b2c1ca8d0 legacy=0x7f8b2c1cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025089450 0 0) 0x7f8b2c0a3410 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient(hunting): my global_id is 4638 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1000c490 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3631622082 0 0) 0x7f8b1000c490 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b2c0a3410 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8b18001c60 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient(hunting): dump: 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:30:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1984218055 0 0) 0x7f8b2c0a3410 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 >> v1:172.21.15.94:6792/0 conn(0x7f8b1000d4b0 legacy=0x7f8b2c133b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 >> v1:172.21.15.94:6789/0 conn(0x7f8b2c1d1330 legacy=0x7f8b2c1d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient: _finish_hunting 0 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 monclient: found mon.g 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c125db0 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient: _finish_auth 0 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient: _check_auth_tickets 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b32b65640 5 monclient: authenticate success, global_id 4638 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b32b65640 10 monclient: _renew_subs 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b32b65640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2c125ff0 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b32b65640 10 monclient: _renew_subs 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b32b65640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2c126d10 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b18004b50 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8b18005000 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b2affd640 4 set_mon_vals no callback set 2024-09-16T10:30:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.228+0000 7f8b297fa640 10 monclient: dump: 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.229+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f8b180051e0 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.229+0000 7f8b32b65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.229+0000 7f8b32b65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.230+0000 7f8b32b65640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.230+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 --> v1:172.21.15.152:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c005040 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.230+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3506+0+0 (unknown 4039528226 0 0) 0x7f8b18045ef0 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.230+0000 7f8b30901640 10 monclient: get_auth_request con 0x7f8b1003a630 auth_method 0 2024-09-16T10:30:02.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.235+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f8b180151d0 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.235+0000 7f8b297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:02.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.235+0000 7f8b297fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:30:02.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.474+0000 7f8b32b65640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:30:02.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.474+0000 7f8b32b65640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:30:02.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.474+0000 7f8b32b65640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:02.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.474+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 --> v1:172.21.15.152:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8b0c005870 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.475+0000 7f8b297fa640 1 -- 172.21.15.94:0/4009944231 <== mon.3 v1:172.21.15.152:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+2706 (unknown 254746657 0 4072516328) 0x7f8b18013870 con 0x7f8b2c1ca8d0 2024-09-16T10:30:02.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.475+0000 7f8b297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:30:02.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.475+0000 7f8b297fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:30:02.478 INFO:teuthology.orchestra.run.smithi094.stdout:{"election_epoch":14,"quorum":[0,1,2,3,4,5,7,8],"quorum_names":["a","f","b","g","c","h","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","modified":"2024-09-16T10:28:46.257482Z","created":"2024-09-16T10:28:46.257482Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6790","nonce":0}]},"addr":"172.21.15.94:6790/0","public_addr":"172.21.15.94:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6790","nonce":0}]},"addr":"172.21.15.152:6790/0","public_addr":"172.21.15.152:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6791","nonce":0}]},"addr":"172.21.15.94:6791/0","public_addr":"172.21.15.94:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6791","nonce":0}]},"addr":"172.21.15.152:6791/0","public_addr":"172.21.15.152:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6792","nonce":0}]},"addr":"172.21.15.94:6792/0","public_addr":"172.21.15.94:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6792","nonce":0}]},"addr":"172.21.15.152:6792/0","public_addr":"172.21.15.152:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6793","nonce":0}]},"addr":"172.21.15.94:6793/0","public_addr":"172.21.15.94:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:30:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.477+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8b1003a630 legacy=0x7f8b1003caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.477+0000 7f8b32b65640 10 monclient: shutdown 2024-09-16T10:30:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.477+0000 7f8b32b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.477+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 >> v1:172.21.15.152:6790/0 conn(0x7f8b2c1ca8d0 legacy=0x7f8b2c1cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.477+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 shutdown_connections 2024-09-16T10:30:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.477+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 >> 172.21.15.94:0/4009944231 conn(0x7f8b2c0a1680 msgr2=0x7f8b2c0a3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.477+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 shutdown_connections 2024-09-16T10:30:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.477+0000 7f8b32b65640 1 -- 172.21.15.94:0/4009944231 wait complete. 2024-09-16T10:30:02.490 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-09-16T10:30:02.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-09-16T10:30:02.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:30:02.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.587+0000 7fbd81812640 10 monclient: get_monmap_and_config 2024-09-16T10:30:02.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.587+0000 7fbd81812640 10 monclient: build_initial_monmap 2024-09-16T10:30:02.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.587+0000 7fbd81812640 10 monclient: monmap: 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:30:02.588706+0000 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:30:02.588706+0000 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:02.591 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:02.591 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:02.591 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:02.591 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:02.591 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.588+0000 7fbd81812640 1 Processor -- start 2024-09-16T10:30:02.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.588+0000 7fbd81812640 1 -- start start 2024-09-16T10:30:02.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.588+0000 7fbd81812640 10 monclient: init 2024-09-16T10:30:02.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.588+0000 7fbd81812640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:02.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.588+0000 7fbd81812640 10 monclient: _add_conns ranks=[5,2,4,3,0,1,6,7,8] 2024-09-16T10:30:02.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd81812640 10 monclient(hunting): picked mon.noname-h con 0x7fbd7c0a3860 addr v1:172.21.15.152:6791/0 2024-09-16T10:30:02.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd81812640 10 monclient(hunting): picked mon.noname-b con 0x7fbd7c06c1d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd81812640 10 monclient(hunting): picked mon.noname-c con 0x7fbd7c0a4350 addr v1:172.21.15.94:6791/0 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd81812640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7c061ae0 con 0x7fbd7c06c1d0 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd81812640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7c0a0600 con 0x7fbd7c0a4350 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd81812640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7c0a0b30 con 0x7fbd7c0a3860 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd81812640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd81812640 10 monclient(hunting): authenticate will time out at 769.929074s 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd7b7fe640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fbd7c0a4350 0x7fbd7c13ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:45756/0 (socket says 172.21.15.94:45756) 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.589+0000 7fbd7b7fe640 1 -- 172.21.15.94:0/1232906841 learned_addr learned my addr 172.21.15.94:0/1232906841 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd7b7fe640 1 -- 172.21.15.94:0/1232906841 >> v1:172.21.15.94:6791/0 conn(0x7fbd7c0a4350 legacy=0x7fbd7c13ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd7a7fc640 1 -- 172.21.15.94:0/1232906841 >> v1:172.21.15.94:6790/0 conn(0x7fbd7c06c1d0 legacy=0x7fbd7c13b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd7affd640 1 -- 172.21.15.94:0/1232906841 >> v1:172.21.15.152:6791/0 conn(0x7fbd7c0a3860 legacy=0x7fbd7c0a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818677876 0 0) 0x7fbd7c0a0600 con 0x7fbd7c0a4350 2024-09-16T10:30:02.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): my global_id is 4522 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd64003520 con 0x7fbd7c0a4350 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829307884 0 0) 0x7fbd7c061ae0 con 0x7fbd7c06c1d0 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): my global_id is 4457 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd7c0a0600 con 0x7fbd7c06c1d0 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373372442 0 0) 0x7fbd7c0a0b30 con 0x7fbd7c0a3860 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient(hunting): my global_id is 4523 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd7c061ae0 con 0x7fbd7c0a3860 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285610841 0 0) 0x7fbd64003520 con 0x7fbd7c0a4350 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 >> v1:172.21.15.152:6791/0 conn(0x7fbd7c0a3860 legacy=0x7fbd7c0a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 >> v1:172.21.15.94:6790/0 conn(0x7fbd7c06c1d0 legacy=0x7fbd7c13b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 monclient: found mon.noname-c 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c0a0e10 con 0x7fbd7c0a4350 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient: _finish_auth 0 2024-09-16T10:30:02.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd74001de0 con 0x7fbd7c0a4350 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd81812640 5 monclient: authenticate success, global_id 4522 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd81812640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.591+0000 7fbd79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 10 monclient: dump: 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd700019d0 con 0x7fbd7c06c1d0 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd740020f0 con 0x7fbd7c0a4350 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 10 monclient: get_monmap_and_config success 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 1 -- 172.21.15.94:0/1232906841 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd74004f60 con 0x7fbd7c0a4350 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 4 set_mon_vals no callback set 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:30:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd79ffb640 10 monclient: dump: 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 10 monclient: shutdown 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 1 -- 172.21.15.94:0/1232906841 >> v1:172.21.15.94:6791/0 conn(0x7fbd7c0a4350 legacy=0x7fbd7c13ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 1 -- 172.21.15.94:0/1232906841 shutdown_connections 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 1 -- 172.21.15.94:0/1232906841 >> 172.21.15.94:0/1232906841 conn(0x7fbd7c0a5960 msgr2=0x7fbd7c0a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 1 -- 172.21.15.94:0/1232906841 shutdown_connections 2024-09-16T10:30:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.592+0000 7fbd81812640 1 -- 172.21.15.94:0/1232906841 wait complete. 2024-09-16T10:30:02.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.593+0000 7fbd81812640 10 monclient: build_initial_monmap 2024-09-16T10:30:02.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.593+0000 7fbd81812640 10 monclient: monmap: 2024-09-16T10:30:02.605 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:02.605 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:30:02.605 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:30:02.605 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:30:02.605 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:02.605 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.605 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:02.606 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:02.606 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:02.606 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:02.606 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:02.606 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:02.606 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:02.606 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:02.606 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:02.606 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.593+0000 7fbd81812640 1 Processor -- start 2024-09-16T10:30:02.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.593+0000 7fbd81812640 1 -- start start 2024-09-16T10:30:02.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.593+0000 7fbd81812640 10 monclient: init 2024-09-16T10:30:02.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:02.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 10 monclient: _add_conns ranks=[7,0,1,6,3,5,2,8,4] 2024-09-16T10:30:02.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 10 monclient(hunting): picked mon.noname-i con 0x7fbd7c06c1d0 addr v1:172.21.15.152:6792/0 2024-09-16T10:30:02.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 10 monclient(hunting): picked mon.noname-a con 0x7fbd7c0a3860 addr v1:172.21.15.94:6789/0 2024-09-16T10:30:02.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 10 monclient(hunting): picked mon.noname-f con 0x7fbd7c0a4350 addr v1:172.21.15.152:6789/0 2024-09-16T10:30:02.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7c1ac130 con 0x7fbd7c0a3860 2024-09-16T10:30:02.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7c09f060 con 0x7fbd7c0a4350 2024-09-16T10:30:02.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7c06b9a0 con 0x7fbd7c06c1d0 2024-09-16T10:30:02.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:02.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd81812640 10 monclient(hunting): authenticate will time out at 769.934219s 2024-09-16T10:30:02.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd7a7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbd7c0a3860 0x7fbd7c1be770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42086/0 (socket says 172.21.15.94:42086) 2024-09-16T10:30:02.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.594+0000 7fbd7a7fc640 1 -- 172.21.15.94:0/1161849533 learned_addr learned my addr 172.21.15.94:0/1161849533 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:02.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd7a7fc640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.94:6789/0 conn(0x7fbd7c0a3860 legacy=0x7fbd7c1be770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd7affd640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.152:6792/0 conn(0x7fbd7c06c1d0 legacy=0x7fbd7c1ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd7b7fe640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.152:6789/0 conn(0x7fbd7c0a4350 legacy=0x7fbd7c1d2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713338922 0 0) 0x7fbd7c1ac130 con 0x7fbd7c0a3860 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): my global_id is 4473 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd68003520 con 0x7fbd7c0a3860 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966584453 0 0) 0x7fbd7c06b9a0 con 0x7fbd7c06c1d0 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): my global_id is 4516 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd7c1ac130 con 0x7fbd7c06c1d0 2024-09-16T10:30:02.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176653989 0 0) 0x7fbd7c09f060 con 0x7fbd7c0a4350 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): my global_id is 4429 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd7c06b9a0 con 0x7fbd7c0a4350 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1661286266 0 0) 0x7fbd68003520 con 0x7fbd7c0a3860 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd7c09f060 con 0x7fbd7c0a3860 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd700014f0 con 0x7fbd7c0a3860 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): dump: 2024-09-16T10:30:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.619 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.596+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2135290411 0 0) 0x7fbd7c09f060 con 0x7fbd7c0a3860 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.152:6792/0 conn(0x7fbd7c06c1d0 legacy=0x7fbd7c1ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.152:6789/0 conn(0x7fbd7c0a4350 legacy=0x7fbd7c1d2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 10 monclient: _finish_hunting 0 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 1 monclient: found mon.a 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c09f5b0 con 0x7fbd7c0a3860 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 10 monclient: _finish_auth 0 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 10 monclient: _check_auth_tickets 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd81812640 5 monclient: authenticate success, global_id 4473 2024-09-16T10:30:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd81812640 10 monclient: _renew_subs 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd81812640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7c1ab890 con 0x7fbd7c0a3860 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd81812640 10 monclient: _renew_subs 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd81812640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7c1bee80 con 0x7fbd7c0a3860 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd70001d10 con 0x7fbd7c0a3860 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd700040f0 con 0x7fbd7c0a3860 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd797fa640 4 set_mon_vals no callback set 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd63fff640 10 monclient: dump: 2024-09-16T10:30:02.621 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.622 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd81812640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.597+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbd5c006580 con 0x7fbd5c002820 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.598+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fbd70004310 con 0x7fbd7c0a3860 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.598+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(20..20 src has 1..20) v4 ==== 3898+0+0 (unknown 541210984 0 0) 0x7fbd70046140 con 0x7fbd7c0a3860 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.598+0000 7fbd7b7fe640 10 monclient: get_auth_request con 0x7fbd680391f0 auth_method 0 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd7affd640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.94:6789/0 conn(0x7fbd5c002820 legacy=0x7fbd5c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695196545 0 0) 0x7fbd5c006580 con 0x7fbd5c002820 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 10 monclient(hunting): my global_id is 4482 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd7c09f060 con 0x7fbd5c002820 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 499450065 0 0) 0x7fbd7c09f060 con 0x7fbd5c002820 2024-09-16T10:30:02.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd5c006580 con 0x7fbd5c002820 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd6c001cb0 con 0x7fbd5c002820 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 10 monclient: dump: 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.624 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3847317095 0 0) 0x7fbd5c006580 con 0x7fbd5c002820 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd5c00a0d0 con 0x7fbd5c002820 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fbd6c0057c0 con 0x7fbd5c002820 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:30:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.94:6789/0 conn(0x7fbd5c002820 legacy=0x7fbd5c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.600+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.94:6789/0 conn(0x7fbd5c002820 legacy=0x7fbd5c004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.606+0000 7fbd81812640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.606+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbd5c009d70 con 0x7fbd5c009090 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd7affd640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.94:6789/0 conn(0x7fbd5c009090 legacy=0x7fbd5c009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074518985 0 0) 0x7fbd5c009d70 con 0x7fbd5c009090 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 10 monclient(hunting): my global_id is 4491 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd5c006580 con 0x7fbd5c009090 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1478096710 0 0) 0x7fbd5c006580 con 0x7fbd5c009090 2024-09-16T10:30:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd5c009d70 con 0x7fbd5c009090 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd6c003fe0 con 0x7fbd5c009090 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.609+0000 7fbd63fff640 10 monclient: dump: 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1856220172 0 0) 0x7fbd5c009d70 con 0x7fbd5c009090 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 --> v1:172.21.15.94:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbd5c00c280 con 0x7fbd5c009090 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 <== mon.0 v1:172.21.15.94:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7443 (unknown 0 0 2186179725) 0x7fbd6c005930 con 0x7fbd5c009090 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd63fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd63fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:30:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.94:6789/0 conn(0x7fbd5c009090 legacy=0x7fbd5c009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd63fff640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.94:6789/0 conn(0x7fbd5c009090 legacy=0x7fbd5c009520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:02.629 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:30:02.629 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:30:02.629 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:30:02.629 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "leader", 2024-09-16T10:30:02.629 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 14, 2024-09-16T10:30:02.629 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 0, 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:30:02.630 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:30:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:30:02.633 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:02.634 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.635 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:30:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.638 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.639 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.641 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.644 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.646 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 3 2024-09-16T10:30:02.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.649 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.649 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:30:02.649 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:30:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.152:6800/2686360781 conn(0x7fbd680391f0 legacy=0x7fbd6803b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd81812640 10 monclient: shutdown 2024-09-16T10:30:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd81812640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 >> v1:172.21.15.94:6789/0 conn(0x7fbd7c0a3860 legacy=0x7fbd7c1be770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd7b7fe640 1 -- 172.21.15.94:0/1161849533 reap_dead start 2024-09-16T10:30:02.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd7b7fe640 1 -- 172.21.15.94:0/1161849533 reap_dead start 2024-09-16T10:30:02.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 shutdown_connections 2024-09-16T10:30:02.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 >> 172.21.15.94:0/1161849533 conn(0x7fbd7c0a5960 msgr2=0x7fbd7c0a75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:02.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 shutdown_connections 2024-09-16T10:30:02.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.610+0000 7fbd81812640 1 -- 172.21.15.94:0/1161849533 wait complete. 2024-09-16T10:30:02.651 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-16T10:30:02.651 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-09-16T10:30:02.651 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:30:02.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.747+0000 7fa3e495e640 10 monclient: get_monmap_and_config 2024-09-16T10:30:02.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.747+0000 7fa3e495e640 10 monclient: build_initial_monmap 2024-09-16T10:30:02.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.747+0000 7fa3e495e640 10 monclient: monmap: 2024-09-16T10:30:02.749 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:02.749 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:30:02.748627+0000 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:30:02.748627+0000 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:02.750 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:02.751 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:02.751 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:02.751 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.748+0000 7fa3e495e640 1 Processor -- start 2024-09-16T10:30:02.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.748+0000 7fa3e495e640 1 -- start start 2024-09-16T10:30:02.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.748+0000 7fa3e495e640 10 monclient: init 2024-09-16T10:30:02.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 10 monclient: _add_conns ranks=[5,6,8,0,3,2,7,4,1] 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 10 monclient(hunting): picked mon.noname-h con 0x7fa3e00a37d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 10 monclient(hunting): picked mon.noname-d con 0x7fa3e00a42c0 addr v1:172.21.15.94:6792/0 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 10 monclient(hunting): picked mon.noname-e con 0x7fa3e00a1740 addr v1:172.21.15.94:6793/0 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e0061ae0 con 0x7fa3e00a37d0 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e00a4890 con 0x7fa3e00a42c0 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e006c490 con 0x7fa3e00a1740 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3e495e640 10 monclient(hunting): authenticate will time out at 770.089179s 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3ddd9b640 1 -- >> v1:172.21.15.94:6792/0 conn(0x7fa3e00a42c0 legacy=0x7fa3e013b7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.94:6792/0 2024-09-16T10:30:02.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3ded9d640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7fa3e00a1740 0x7fa3e013ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:52042/0 (socket says 172.21.15.94:52042) 2024-09-16T10:30:02.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.749+0000 7fa3ded9d640 1 -- 172.21.15.94:0/1210898980 learned_addr learned my addr 172.21.15.94:0/1210898980 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:02.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3ded9d640 1 -- 172.21.15.94:0/1210898980 >> v1:172.21.15.94:6793/0 conn(0x7fa3e00a1740 legacy=0x7fa3e013ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 -- 172.21.15.94:0/1210898980 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279956862 0 0) 0x7fa3e006c490 con 0x7fa3e00a1740 2024-09-16T10:30:02.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient(hunting): my global_id is 4463 2024-09-16T10:30:02.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 -- 172.21.15.94:0/1210898980 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c8003520 con 0x7fa3e00a1740 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3de59c640 1 -- 172.21.15.94:0/1210898980 >> v1:172.21.15.152:6791/0 conn(0x7fa3e00a37d0 legacy=0x7fa3e00a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 -- 172.21.15.94:0/1210898980 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552361423 0 0) 0x7fa3c8003520 con 0x7fa3e00a1740 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 -- 172.21.15.94:0/1210898980 >> v1:172.21.15.94:6792/0 conn(0x7fa3e00a42c0 legacy=0x7fa3e013b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 -- 172.21.15.94:0/1210898980 >> v1:172.21.15.152:6791/0 conn(0x7fa3e00a37d0 legacy=0x7fa3e00a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient: _finish_hunting 0 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 monclient: found mon.noname-e 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 -- 172.21.15.94:0/1210898980 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e00a0d10 con 0x7fa3e00a1740 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient: _finish_auth 0 2024-09-16T10:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient: _check_auth_tickets 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3e495e640 5 monclient: authenticate success, global_id 4463 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 -- 172.21.15.94:0/1210898980 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fa3d8001de0 con 0x7fa3e00a1740 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient: dump: 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3e495e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 -- 172.21.15.94:0/1210898980 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3d80020f0 con 0x7fa3e00a1740 2024-09-16T10:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.751+0000 7fa3dd59a640 1 -- 172.21.15.94:0/1210898980 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fa3d8004fa0 con 0x7fa3e00a1740 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3e495e640 10 monclient: get_monmap_and_config success 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3e495e640 4 set_mon_vals no callback set 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3dd59a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3dd59a640 10 monclient: dump: 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.764 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3e495e640 10 monclient: shutdown 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3e495e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3e495e640 1 -- 172.21.15.94:0/1210898980 >> v1:172.21.15.94:6793/0 conn(0x7fa3e00a1740 legacy=0x7fa3e013ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3e495e640 1 -- 172.21.15.94:0/1210898980 shutdown_connections 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3e495e640 1 -- 172.21.15.94:0/1210898980 >> 172.21.15.94:0/1210898980 conn(0x7fa3e00a9460 msgr2=0x7fa3e00a58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:02.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3e495e640 1 -- 172.21.15.94:0/1210898980 shutdown_connections 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.752+0000 7fa3e495e640 1 -- 172.21.15.94:0/1210898980 wait complete. 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.753+0000 7fa3e495e640 10 monclient: build_initial_monmap 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.753+0000 7fa3e495e640 10 monclient: monmap: 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.753+0000 7fa3e495e640 1 Processor -- start 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.753+0000 7fa3e495e640 1 -- start start 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.753+0000 7fa3e495e640 10 monclient: init 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 10 monclient: _add_conns ranks=[3,5,2,4,0,6,7,1,8] 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 10 monclient(hunting): picked mon.noname-g con 0x7fa3e00a1740 addr v1:172.21.15.152:6790/0 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 10 monclient(hunting): picked mon.noname-h con 0x7fa3e00a37d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e00a42c0 addr v1:172.21.15.94:6790/0 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e009f6b0 con 0x7fa3e00a42c0 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e01a9720 con 0x7fa3e00a1740 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e00a7330 con 0x7fa3e00a37d0 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3e495e640 10 monclient(hunting): authenticate will time out at 770.094179s 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3ded9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fa3e00a42c0 0x7fa3e01d3570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:54262/0 (socket says 172.21.15.94:54262) 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.754+0000 7fa3ded9d640 1 -- 172.21.15.94:0/2995536037 learned_addr learned my addr 172.21.15.94:0/2995536037 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ded9d640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.94:6790/0 conn(0x7fa3e00a42c0 legacy=0x7fa3e01d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132281508 0 0) 0x7fa3e009f6b0 con 0x7fa3e00a42c0 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): my global_id is 4466 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ddd9b640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.152:6791/0 conn(0x7fa3e00a37d0 legacy=0x7fa3e01cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c0003520 con 0x7fa3e00a42c0 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3de59c640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.152:6790/0 conn(0x7fa3e00a1740 legacy=0x7fa3e01be8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126807766 0 0) 0x7fa3e01a9720 con 0x7fa3e00a1740 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): my global_id is 4647 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e009f6b0 con 0x7fa3e00a1740 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338292379 0 0) 0x7fa3e00a7330 con 0x7fa3e00a37d0 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): my global_id is 4541 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e01a9720 con 0x7fa3e00a37d0 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4028997001 0 0) 0x7fa3c0003520 con 0x7fa3e00a42c0 2024-09-16T10:30:02.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3e00a7330 con 0x7fa3e00a42c0 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fa3d8004ea0 con 0x7fa3e00a42c0 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): dump: 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.772 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1952022339 0 0) 0x7fa3e00a7330 con 0x7fa3e00a42c0 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.152:6791/0 conn(0x7fa3e00a37d0 legacy=0x7fa3e01cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.152:6790/0 conn(0x7fa3e00a1740 legacy=0x7fa3e01be8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient: _finish_hunting 0 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 monclient: found mon.b 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.756+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e00a7880 con 0x7fa3e00a42c0 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 10 monclient: _finish_auth 0 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 10 monclient: _check_auth_tickets 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3e495e640 5 monclient: authenticate success, global_id 4466 2024-09-16T10:30:02.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3e495e640 10 monclient: _renew_subs 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3e495e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e01ac270 con 0x7fa3e00a42c0 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3e495e640 10 monclient: _renew_subs 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3e495e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e01a9f10 con 0x7fa3e00a42c0 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3d80050c0 con 0x7fa3e00a42c0 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fa3d8001900 con 0x7fa3e00a42c0 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:30:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3dcd99640 4 set_mon_vals no callback set 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 10 monclient: dump: 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.775 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.776 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.776 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.776 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.776 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3e495e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:02.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa3c4006580 con 0x7fa3c4002820 2024-09-16T10:30:02.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.757+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fa3d8010810 con 0x7fa3e00a42c0 2024-09-16T10:30:02.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.758+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(20..20 src has 1..20) v4 ==== 3898+0+0 (unknown 541210984 0 0) 0x7fa3d8046550 con 0x7fa3e00a42c0 2024-09-16T10:30:02.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.758+0000 7fa3ddd9b640 10 monclient: get_auth_request con 0x7fa3c00392a0 auth_method 0 2024-09-16T10:30:02.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3de59c640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.94:6790/0 conn(0x7fa3c4002820 legacy=0x7fa3c4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310295605 0 0) 0x7fa3c4006580 con 0x7fa3c4002820 2024-09-16T10:30:02.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 10 monclient(hunting): my global_id is 4475 2024-09-16T10:30:02.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e00a7330 con 0x7fa3c4002820 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3112630144 0 0) 0x7fa3e00a7330 con 0x7fa3c4002820 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3c4006580 con 0x7fa3c4002820 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fa3d0001bf0 con 0x7fa3c4002820 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 10 monclient: dump: 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.780 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3207013728 0 0) 0x7fa3c4006580 con 0x7fa3c4002820 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c400a0d0 con 0x7fa3c4002820 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa3d00057c0 con 0x7fa3c4002820 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.94:6790/0 conn(0x7fa3c4002820 legacy=0x7fa3c4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.760+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.94:6790/0 conn(0x7fa3c4002820 legacy=0x7fa3c4004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.766+0000 7fa3e495e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.766+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa3c4009d70 con 0x7fa3c4009090 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3de59c640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.94:6790/0 conn(0x7fa3c4009090 legacy=0x7fa3c4009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20092772 0 0) 0x7fa3c4009d70 con 0x7fa3c4009090 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 10 monclient(hunting): my global_id is 4484 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c4006580 con 0x7fa3c4009090 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 320745004 0 0) 0x7fa3c4006580 con 0x7fa3c4009090 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3c4009d70 con 0x7fa3c4009090 2024-09-16T10:30:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fa3d0003ec0 con 0x7fa3c4009090 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 10 monclient: dump: 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1755308633 0 0) 0x7fa3c4009d70 con 0x7fa3c4009090 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 --> v1:172.21.15.94:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa3c400c280 con 0x7fa3c4009090 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 <== mon.2 v1:172.21.15.94:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7773 (unknown 0 0 592822327) 0x7fa3d0004430 con 0x7fa3c4009090 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:30:02.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.94:6790/0 conn(0x7fa3c4009090 legacy=0x7fa3c4009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.769+0000 7fa3ceffd640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.94:6790/0 conn(0x7fa3c4009090 legacy=0x7fa3c4009520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:02.785 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:30:02.785 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:30:02.785 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:30:02.785 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:30:02.785 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 14, 2024-09-16T10:30:02.785 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:30:02.785 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:30:02.785 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 0, 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:02.786 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:30:02.787 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:30:02.789 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:02.790 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:30:02.791 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:30:02.792 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.793 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.794 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:30:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:30:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:30:02.799 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.801 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.802 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.803 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.804 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:30:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": [ 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:30:02.806 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.152:6800/2686360781 conn(0x7fa3c00392a0 legacy=0x7fa3c003b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3e495e640 10 monclient: shutdown 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3e495e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 >> v1:172.21.15.94:6790/0 conn(0x7fa3e00a42c0 legacy=0x7fa3e01d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3ded9d640 1 -- 172.21.15.94:0/2995536037 reap_dead start 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3ded9d640 1 -- 172.21.15.94:0/2995536037 reap_dead start 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 shutdown_connections 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 >> 172.21.15.94:0/2995536037 conn(0x7fa3e00a9460 msgr2=0x7fa3e00a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 shutdown_connections 2024-09-16T10:30:02.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.770+0000 7fa3e495e640 1 -- 172.21.15.94:0/2995536037 wait complete. 2024-09-16T10:30:02.809 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-16T10:30:02.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-09-16T10:30:02.809 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:30:02.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.905+0000 7f25d26d0640 10 monclient: get_monmap_and_config 2024-09-16T10:30:02.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.905+0000 7f25d26d0640 10 monclient: build_initial_monmap 2024-09-16T10:30:02.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.905+0000 7f25d26d0640 10 monclient: monmap: 2024-09-16T10:30:02.907 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:02.907 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.907 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:30:02.906580+0000 2024-09-16T10:30:02.907 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:30:02.906580+0000 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:02.908 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.906+0000 7f25d26d0640 1 Processor -- start 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.906+0000 7f25d26d0640 1 -- start start 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.906+0000 7f25d26d0640 10 monclient: init 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.906+0000 7f25d26d0640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25d26d0640 10 monclient: _add_conns ranks=[2,5,3,0,6,1,8,4,7] 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25d26d0640 10 monclient(hunting): picked mon.noname-b con 0x7f25cc0a37d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25d26d0640 10 monclient(hunting): picked mon.noname-h con 0x7f25cc0a42c0 addr v1:172.21.15.152:6791/0 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25d26d0640 10 monclient(hunting): picked mon.noname-g con 0x7f25cc0a1740 addr v1:172.21.15.152:6790/0 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25d26d0640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc061ae0 con 0x7f25cc0a37d0 2024-09-16T10:30:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25d26d0640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc0a4890 con 0x7f25cc0a1740 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25d26d0640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc06c490 con 0x7f25cc0a42c0 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25d26d0640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25d26d0640 10 monclient(hunting): authenticate will time out at 770.247105s 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25cbfff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f25cc0a37d0 0x7f25cc0a3bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:54288/0 (socket says 172.21.15.94:54288) 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.907+0000 7f25cbfff640 1 -- 172.21.15.94:0/3162141351 learned_addr learned my addr 172.21.15.94:0/3162141351 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25cbfff640 1 -- 172.21.15.94:0/3162141351 >> v1:172.21.15.94:6790/0 conn(0x7f25cc0a37d0 legacy=0x7f25cc0a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25d0c6d640 1 -- 172.21.15.94:0/3162141351 >> v1:172.21.15.152:6790/0 conn(0x7f25cc0a1740 legacy=0x7f25cc13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25cb7fe640 1 -- 172.21.15.94:0/3162141351 >> v1:172.21.15.152:6791/0 conn(0x7f25cc0a42c0 legacy=0x7f25cc13b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583700795 0 0) 0x7f25cc061ae0 con 0x7f25cc0a37d0 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): my global_id is 4493 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b4003520 con 0x7f25cc0a37d0 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254463772 0 0) 0x7f25cc06c490 con 0x7f25cc0a42c0 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): my global_id is 4550 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc061ae0 con 0x7f25cc0a42c0 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801043856 0 0) 0x7f25cc0a4890 con 0x7f25cc0a1740 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient(hunting): my global_id is 4656 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc06c490 con 0x7f25cc0a1740 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1789308038 0 0) 0x7f25b4003520 con 0x7f25cc0a37d0 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 >> v1:172.21.15.152:6791/0 conn(0x7f25cc0a42c0 legacy=0x7f25cc13b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 >> v1:172.21.15.152:6790/0 conn(0x7f25cc0a1740 legacy=0x7f25cc13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: _finish_hunting 0 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 monclient: found mon.noname-b 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc0a0d10 con 0x7f25cc0a37d0 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: _finish_auth 0 2024-09-16T10:30:02.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: _check_auth_tickets 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25bc001a60 con 0x7f25cc0a37d0 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25d26d0640 5 monclient: authenticate success, global_id 4493 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25d26d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: dump: 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25d26d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25bc001d70 con 0x7f25cc0a37d0 2024-09-16T10:30:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25d26d0640 10 monclient: get_monmap_and_config success 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 1 -- 172.21.15.94:0/3162141351 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25bc004be0 con 0x7f25cc0a37d0 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25d26d0640 4 set_mon_vals no callback set 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.909+0000 7f25caffd640 10 monclient: dump: 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.920 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.910+0000 7f25d26d0640 10 monclient: shutdown 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.910+0000 7f25d26d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:02.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.910+0000 7f25d26d0640 1 -- 172.21.15.94:0/3162141351 >> v1:172.21.15.94:6790/0 conn(0x7f25cc0a37d0 legacy=0x7f25cc0a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.910+0000 7f25d26d0640 1 -- 172.21.15.94:0/3162141351 shutdown_connections 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.910+0000 7f25d26d0640 1 -- 172.21.15.94:0/3162141351 >> 172.21.15.94:0/3162141351 conn(0x7f25cc0a9460 msgr2=0x7f25cc0a58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.910+0000 7f25d26d0640 1 -- 172.21.15.94:0/3162141351 shutdown_connections 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.910+0000 7f25d26d0640 1 -- 172.21.15.94:0/3162141351 wait complete. 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 10 monclient: build_initial_monmap 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 10 monclient: monmap: 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:02.922 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 1 Processor -- start 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 1 -- start start 2024-09-16T10:30:02.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 10 monclient: init 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 10 monclient: _add_conns ranks=[3,8,7,4,1,0,2,6,5] 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 10 monclient(hunting): picked mon.noname-g con 0x7f25cc0a1740 addr v1:172.21.15.152:6790/0 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 10 monclient(hunting): picked mon.noname-e con 0x7f25cc0a37d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.911+0000 7f25d26d0640 10 monclient(hunting): picked mon.noname-i con 0x7f25cc0a42c0 addr v1:172.21.15.152:6792/0 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.912+0000 7f25d26d0640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc0a79c0 con 0x7f25cc0a1740 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.912+0000 7f25d26d0640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc0a7470 con 0x7f25cc0a42c0 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.912+0000 7f25d26d0640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc09f700 con 0x7f25cc0a37d0 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.912+0000 7f25d26d0640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.912+0000 7f25d26d0640 10 monclient(hunting): authenticate will time out at 770.252040s 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.912+0000 7f25cb7fe640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f25cc0a37d0 0x7f25cc1cfd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:52058/0 (socket says 172.21.15.94:52058) 2024-09-16T10:30:02.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.912+0000 7f25cb7fe640 1 -- 172.21.15.94:0/2847931327 learned_addr learned my addr 172.21.15.94:0/2847931327 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:02.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25cb7fe640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.94:6793/0 conn(0x7f25cc0a37d0 legacy=0x7f25cc1cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25cbfff640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.152:6790/0 conn(0x7f25cc0a1740 legacy=0x7f25cc1bd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25d0c6d640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.152:6792/0 conn(0x7f25cc0a42c0 legacy=0x7f25cc1d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25cb7fe640 1 --1- 172.21.15.94:0/2847931327 >> v1:172.21.15.94:6793/0 conn(0x7f25cc0a37d0 0x7f25cc1cfd90 :-1 s=OPENED pgs=85 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7f25cc09f700 auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394902730 0 0) 0x7f25cc0a79c0 con 0x7f25cc0a1740 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): my global_id is 4665 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b8003520 con 0x7f25cc0a1740 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672592065 0 0) 0x7f25cc0a7470 con 0x7f25cc0a42c0 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): my global_id is 4525 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc0a79c0 con 0x7f25cc0a42c0 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 557043222 0 0) 0x7f25b8003520 con 0x7f25cc0a1740 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25cc0a7470 con 0x7f25cc0a1740 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25bc004b20 con 0x7f25cc0a1740 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): dump: 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.932 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.933 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1832440767 0 0) 0x7f25cc0a79c0 con 0x7f25cc0a42c0 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25b8003520 con 0x7f25cc0a42c0 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25c40020d0 con 0x7f25cc0a42c0 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient(hunting): dump: 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1842660136 0 0) 0x7f25cc0a7470 con 0x7f25cc0a1740 2024-09-16T10:30:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.94:6793/0 conn(0x7f25cc0a37d0 legacy=0x7f25cc1cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.152:6792/0 conn(0x7f25cc0a42c0 legacy=0x7f25cc1d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient: _finish_hunting 0 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 monclient: found mon.g 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc1a8d00 con 0x7f25cc0a1740 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient: _finish_auth 0 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient: _check_auth_tickets 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912705957 0 0) 0x7f25cc09f700 con 0x7f25cc0a37d0 2024-09-16T10:30:02.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25d26d0640 5 monclient: authenticate success, global_id 4665 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25d26d0640 10 monclient: _renew_subs 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25d26d0640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25cc1aad80 con 0x7f25cc0a1740 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.914+0000 7f25c8ff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25d26d0640 10 monclient: _renew_subs 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25d26d0640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25cc1bddb0 con 0x7f25cc0a1740 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25bc001320 con 0x7f25cc0a1740 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25bc005550 con 0x7f25cc0a1740 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25ca7fc640 4 set_mon_vals no callback set 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25c8ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25c8ff9640 10 monclient: dump: 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25d26d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f25ac006580 con 0x7f25ac002820 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.915+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f25bc0105b0 con 0x7f25cc0a1740 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.916+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(20..20 src has 1..20) v4 ==== 3898+0+0 (unknown 541210984 0 0) 0x7f25bc046310 con 0x7f25cc0a1740 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.916+0000 7f25d0c6d640 10 monclient: get_auth_request con 0x7f25b80396e0 auth_method 0 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.917+0000 7f25cb7fe640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.94:6791/0 conn(0x7f25ac002820 legacy=0x7f25ac004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637770107 0 0) 0x7f25ac006580 con 0x7f25ac002820 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 10 monclient(hunting): my global_id is 4531 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc09f700 con 0x7f25ac002820 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 462340004 0 0) 0x7f25cc09f700 con 0x7f25ac002820 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25ac006580 con 0x7f25ac002820 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25c0001d20 con 0x7f25ac002820 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 10 monclient: dump: 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.940 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1975948887 0 0) 0x7f25ac006580 con 0x7f25ac002820 2024-09-16T10:30:02.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.94:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25ac00a0d0 con 0x7f25ac002820 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f25c00057c0 con 0x7f25ac002820 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.94:6791/0 conn(0x7f25ac002820 legacy=0x7f25ac004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.918+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.94:6791/0 conn(0x7f25ac002820 legacy=0x7f25ac004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.924+0000 7f25d26d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.924+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f25ac009d90 con 0x7f25ac0090b0 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25cb7fe640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.94:6791/0 conn(0x7f25ac0090b0 legacy=0x7f25ac009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803139156 0 0) 0x7f25ac009d90 con 0x7f25ac0090b0 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:02.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 10 monclient(hunting): my global_id is 4540 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac006580 con 0x7f25ac0090b0 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2155150240 0 0) 0x7f25ac006580 con 0x7f25ac0090b0 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25ac009d90 con 0x7f25ac0090b0 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25c0004080 con 0x7f25ac0090b0 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 10 monclient: dump: 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:02.943 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:02.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 111474050 0 0) 0x7f25ac009d90 con 0x7f25ac0090b0 2024-09-16T10:30:02.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 --> v1:172.21.15.94:6791/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f25ac00c280 con 0x7f25ac0090b0 2024-09-16T10:30:02.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 <== mon.4 v1:172.21.15.94:6791/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7441 (unknown 0 0 3479709355) 0x7f25c0005930 con 0x7f25ac0090b0 2024-09-16T10:30:02.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:30:02.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:30:02.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.94:6791/0 conn(0x7f25ac0090b0 legacy=0x7f25ac009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.927+0000 7f25c8ff9640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.94:6791/0 conn(0x7f25ac0090b0 legacy=0x7f25ac009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:02.945 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:30:02.945 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:30:02.945 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 14, 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.946 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 0, 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:02.947 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:02.948 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:30:02.949 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:02.950 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.951 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:30:02.954 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.956 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:30:02.957 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.958 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:30:02.959 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.960 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:30:02.961 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:30:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:02.963 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:02.964 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:30:02.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:02.965 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:02.965 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:02.965 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:02.965 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:02.965 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:02.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:02.965 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:30:02.965 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.152:6800/2686360781 conn(0x7f25b80396e0 legacy=0x7f25b803bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d26d0640 10 monclient: shutdown 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d26d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 >> v1:172.21.15.152:6790/0 conn(0x7f25cc0a1740 legacy=0x7f25cc1bd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d0c6d640 1 -- 172.21.15.94:0/2847931327 reap_dead start 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d0c6d640 1 -- 172.21.15.94:0/2847931327 reap_dead start 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 shutdown_connections 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 >> 172.21.15.94:0/2847931327 conn(0x7f25cc0a9460 msgr2=0x7f25cc0a1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 shutdown_connections 2024-09-16T10:30:02.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:02.928+0000 7f25d26d0640 1 -- 172.21.15.94:0/2847931327 wait complete. 2024-09-16T10:30:02.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-16T10:30:02.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-09-16T10:30:02.968 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.065+0000 7fb8bfeb1640 10 monclient: get_monmap_and_config 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.065+0000 7fb8bfeb1640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.065+0000 7fb8bfeb1640 10 monclient: monmap: 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:30:03.066207+0000 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:30:03.066207+0000 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.067 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.068 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.068 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.068 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.068 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.068 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.068 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.068 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.068 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.068 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 1 Processor -- start 2024-09-16T10:30:03.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 1 -- start start 2024-09-16T10:30:03.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 10 monclient: init 2024-09-16T10:30:03.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 10 monclient: _add_conns ranks=[5,6,2,8,7,3,4,0,1] 2024-09-16T10:30:03.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 10 monclient(hunting): picked mon.noname-h con 0x7fb8b80a3860 addr v1:172.21.15.152:6791/0 2024-09-16T10:30:03.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 10 monclient(hunting): picked mon.noname-d con 0x7fb8b806c1d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:30:03.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 10 monclient(hunting): picked mon.noname-b con 0x7fb8b80a4350 addr v1:172.21.15.94:6790/0 2024-09-16T10:30:03.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8061ae0 con 0x7fb8b80a4350 2024-09-16T10:30:03.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80a0600 con 0x7fb8b80a3860 2024-09-16T10:30:03.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80a0b30 con 0x7fb8b806c1d0 2024-09-16T10:30:03.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bd44c640 1 -- >> v1:172.21.15.94:6792/0 conn(0x7fb8b806c1d0 legacy=0x7fb8b813b860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.94:6792/0 2024-09-16T10:30:03.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8bfeb1640 10 monclient(hunting): authenticate will time out at 770.406831s 2024-09-16T10:30:03.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8be44e640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fb8b80a4350 0x7fb8b813ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:54294/0 (socket says 172.21.15.94:54294) 2024-09-16T10:30:03.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.066+0000 7fb8be44e640 1 -- 172.21.15.94:0/2372402298 learned_addr learned my addr 172.21.15.94:0/2372402298 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8be44e640 1 -- 172.21.15.94:0/2372402298 >> v1:172.21.15.94:6790/0 conn(0x7fb8b80a4350 legacy=0x7fb8b813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 -- 172.21.15.94:0/2372402298 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745372382 0 0) 0x7fb8b8061ae0 con 0x7fb8b80a4350 2024-09-16T10:30:03.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient(hunting): my global_id is 4502 2024-09-16T10:30:03.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bdc4d640 1 -- 172.21.15.94:0/2372402298 >> v1:172.21.15.152:6791/0 conn(0x7fb8b80a3860 legacy=0x7fb8b80a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 -- 172.21.15.94:0/2372402298 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a0003520 con 0x7fb8b80a4350 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 -- 172.21.15.94:0/2372402298 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 299832191 0 0) 0x7fb8a0003520 con 0x7fb8b80a4350 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 -- 172.21.15.94:0/2372402298 >> v1:172.21.15.94:6792/0 conn(0x7fb8b806c1d0 legacy=0x7fb8b813b860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 -- 172.21.15.94:0/2372402298 >> v1:172.21.15.152:6791/0 conn(0x7fb8b80a3860 legacy=0x7fb8b80a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 monclient: found mon.noname-b 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 -- 172.21.15.94:0/2372402298 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80a0e10 con 0x7fb8b80a4350 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 -- 172.21.15.94:0/2372402298 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb8a8001dd0 con 0x7fb8b80a4350 2024-09-16T10:30:03.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bfeb1640 5 monclient: authenticate success, global_id 4502 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bfeb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: dump: 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.082 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.083 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.083 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.083 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.083 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.083 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.083 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bfeb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:03.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 -- 172.21.15.94:0/2372402298 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8a80020e0 con 0x7fb8b80a4350 2024-09-16T10:30:03.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 1 -- 172.21.15.94:0/2372402298 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb8a8004f60 con 0x7fb8b80a4350 2024-09-16T10:30:03.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bfeb1640 10 monclient: get_monmap_and_config success 2024-09-16T10:30:03.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bfeb1640 4 set_mon_vals no callback set 2024-09-16T10:30:03.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:30:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.069+0000 7fb8bcc4b640 10 monclient: dump: 2024-09-16T10:30:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.087 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.070+0000 7fb8bfeb1640 10 monclient: shutdown 2024-09-16T10:30:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.070+0000 7fb8bfeb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.070+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/2372402298 >> v1:172.21.15.94:6790/0 conn(0x7fb8b80a4350 legacy=0x7fb8b813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.070+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/2372402298 shutdown_connections 2024-09-16T10:30:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.070+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/2372402298 >> 172.21.15.94:0/2372402298 conn(0x7fb8b80a5960 msgr2=0x7fb8b80a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.070+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/2372402298 shutdown_connections 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.070+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/2372402298 wait complete. 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.070+0000 7fb8bfeb1640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.070+0000 7fb8bfeb1640 10 monclient: monmap: 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 1 Processor -- start 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 1 -- start start 2024-09-16T10:30:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 10 monclient: init 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 10 monclient: _add_conns ranks=[3,1,0,4,7,5,8,2,6] 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 10 monclient(hunting): picked mon.noname-g con 0x7fb8b806c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 10 monclient(hunting): picked mon.noname-f con 0x7fb8b80a3860 addr v1:172.21.15.152:6789/0 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 10 monclient(hunting): picked mon.noname-a con 0x7fb8b80a4350 addr v1:172.21.15.94:6789/0 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b806ba70 con 0x7fb8b80a4350 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b81cb810 con 0x7fb8b80a3860 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b809f0d0 con 0x7fb8b806c1d0 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8bfeb1640 10 monclient(hunting): authenticate will time out at 770.411644s 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8be44e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb8b80a4350 0x7fb8b81d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42112/0 (socket says 172.21.15.94:42112) 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.071+0000 7fb8be44e640 1 -- 172.21.15.94:0/430514669 learned_addr learned my addr 172.21.15.94:0/430514669 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.073+0000 7fb8be44e640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.94:6789/0 conn(0x7fb8b80a4350 legacy=0x7fb8b81d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.073+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962753896 0 0) 0x7fb8b806ba70 con 0x7fb8b80a4350 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.073+0000 7fb8b67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.073+0000 7fb8b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.073+0000 7fb8b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.073+0000 7fb8b67fc640 10 monclient(hunting): my global_id is 4500 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.073+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a4003520 con 0x7fb8b80a4350 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bd44c640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.152:6789/0 conn(0x7fb8b80a3860 legacy=0x7fb8b81cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bdc4d640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.152:6790/0 conn(0x7fb8b806c1d0 legacy=0x7fb8b81bd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1002594079 0 0) 0x7fb8a4003520 con 0x7fb8b80a4350 2024-09-16T10:30:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8b806ba70 con 0x7fb8b80a4350 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb8a8001e30 con 0x7fb8b80a4350 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient(hunting): dump: 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.092 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.093 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304859878 0 0) 0x7fb8b809f0d0 con 0x7fb8b806c1d0 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient(hunting): my global_id is 4674 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a4003520 con 0x7fb8b806c1d0 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bd44c640 1 --1- 172.21.15.94:0/430514669 >> v1:172.21.15.152:6789/0 conn(0x7fb8b80a3860 0x7fb8b81cfcc0 :-1 s=OPENED pgs=84 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7fb8b81cb810 auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4278430270 0 0) 0x7fb8b806ba70 con 0x7fb8b80a4350 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.152:6790/0 conn(0x7fb8b806c1d0 legacy=0x7fb8b81bd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.152:6789/0 conn(0x7fb8b80a3860 legacy=0x7fb8b81cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 monclient: found mon.a 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b81aaf90 con 0x7fb8b80a4350 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bfeb1640 5 monclient: authenticate success, global_id 4500 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bfeb1640 10 monclient: _renew_subs 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324842013 0 0) 0x7fb8b81cb810 con 0x7fb8b80a3860 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bfeb1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b81aa6f0 con 0x7fb8b80a4350 2024-09-16T10:30:03.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bfeb1640 10 monclient: _renew_subs 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bfeb1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b81a8c30 con 0x7fb8b80a4350 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8a80020c0 con 0x7fb8b80a4350 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb8a80055e0 con 0x7fb8b80a4350 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b7fff640 4 set_mon_vals no callback set 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:30:03.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.074+0000 7fb8b67fc640 10 monclient: dump: 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.097 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.098 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.098 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.098 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.098 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.075+0000 7fb8bfeb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.075+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb898006580 con 0x7fb898002820 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.075+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fb8a80057c0 con 0x7fb8b80a4350 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.075+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(20..20 src has 1..20) v4 ==== 3898+0+0 (unknown 541210984 0 0) 0x7fb8a8046450 con 0x7fb8b80a4350 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.075+0000 7fb8bd44c640 10 monclient: get_auth_request con 0x7fb8a4038f00 auth_method 0 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8bdc4d640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.94:6793/0 conn(0x7fb898002820 legacy=0x7fb898004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091228484 0 0) 0x7fb898006580 con 0x7fb898002820 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 10 monclient(hunting): my global_id is 4481 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b81cb810 con 0x7fb898002820 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 97321985 0 0) 0x7fb8b81cb810 con 0x7fb898002820 2024-09-16T10:30:03.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb898006580 con 0x7fb898002820 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb8b0001bf0 con 0x7fb898002820 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 10 monclient: dump: 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3644844156 0 0) 0x7fb898006580 con 0x7fb898002820 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.077+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb89800a0d0 con 0x7fb898002820 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.078+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb8b0006810 con 0x7fb898002820 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.078+0000 7fb8b67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.078+0000 7fb8b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:30:03.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.078+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.94:6793/0 conn(0x7fb898002820 legacy=0x7fb898004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.078+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.94:6793/0 conn(0x7fb898002820 legacy=0x7fb898004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.084+0000 7fb8bfeb1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:30:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.084+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb898009d70 con 0x7fb898009090 2024-09-16T10:30:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8bdc4d640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.94:6793/0 conn(0x7fb898009090 legacy=0x7fb898009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017215136 0 0) 0x7fb898009d70 con 0x7fb898009090 2024-09-16T10:30:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 10 monclient(hunting): my global_id is 4490 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb898006580 con 0x7fb898009090 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4169224843 0 0) 0x7fb898006580 con 0x7fb898009090 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb898009d70 con 0x7fb898009090 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb8b0003e70 con 0x7fb898009090 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 10 monclient: dump: 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.104 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.106 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 271788605 0 0) 0x7fb898009d70 con 0x7fb898009090 2024-09-16T10:30:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.086+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 --> v1:172.21.15.94:6793/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb89800c280 con 0x7fb898009090 2024-09-16T10:30:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 <== mon.8 v1:172.21.15.94:6793/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7607 (unknown 0 0 4086794093) 0x7fb8b0004460 con 0x7fb898009090 2024-09-16T10:30:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8b67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:30:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:30:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.94:6793/0 conn(0x7fb898009090 legacy=0x7fb898009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8b67fc640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.94:6793/0 conn(0x7fb898009090 legacy=0x7fb898009520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.106 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 14, 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:30:03.107 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 0, 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.108 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.109 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.110 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.110 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.110 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.110 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.111 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:30:03.111 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:30:03.111 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:30:03.111 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:30:03.111 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:30:03.111 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.112 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:30:03.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.114 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.116 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.119 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:30:03.120 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.121 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.122 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.123 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.124 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.126 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:30:03.127 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:30:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.152:6800/2686360781 conn(0x7fb8a4038f00 legacy=0x7fb8a403b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8bfeb1640 10 monclient: shutdown 2024-09-16T10:30:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8bfeb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 >> v1:172.21.15.94:6789/0 conn(0x7fb8b80a4350 legacy=0x7fb8b81d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8be44e640 1 -- 172.21.15.94:0/430514669 reap_dead start 2024-09-16T10:30:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8be44e640 1 -- 172.21.15.94:0/430514669 reap_dead start 2024-09-16T10:30:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 shutdown_connections 2024-09-16T10:30:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 >> 172.21.15.94:0/430514669 conn(0x7fb8b80a5960 msgr2=0x7fb8b80a7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 shutdown_connections 2024-09-16T10:30:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.087+0000 7fb8bfeb1640 1 -- 172.21.15.94:0/430514669 wait complete. 2024-09-16T10:30:03.129 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-16T10:30:03.129 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-09-16T10:30:03.130 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-16T10:30:03.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.226+0000 7f47e7f9c640 10 monclient: get_monmap_and_config 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.226+0000 7f47e7f9c640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.226+0000 7f47e7f9c640 10 monclient: monmap: 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:30:03.226996+0000 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:30:03.226996+0000 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.228 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.229 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.229 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.229 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.229 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.229 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.226+0000 7f47e7f9c640 1 Processor -- start 2024-09-16T10:30:03.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 1 -- start start 2024-09-16T10:30:03.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 10 monclient: init 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 10 monclient: _add_conns ranks=[1,0,8,7,4,6,3,5,2] 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 10 monclient(hunting): picked mon.noname-f con 0x7f47e01368e0 addr v1:172.21.15.152:6789/0 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 10 monclient(hunting): picked mon.noname-a con 0x7f47e0136cc0 addr v1:172.21.15.94:6789/0 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 10 monclient(hunting): picked mon.noname-e con 0x7f47e01370a0 addr v1:172.21.15.94:6793/0 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e0061ae0 con 0x7f47e0136cc0 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e00a03e0 con 0x7f47e01368e0 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e006c8d0 con 0x7f47e01370a0 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e7f9c640 10 monclient(hunting): authenticate will time out at 770.567636s 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e5537640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f47e0136cc0 0x7f47e013b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42114/0 (socket says 172.21.15.94:42114) 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e6539640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f47e01370a0 0x7f47e013ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:52078/0 (socket says 172.21.15.94:52078) 2024-09-16T10:30:03.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.227+0000 7f47e5537640 1 -- 172.21.15.94:0/1107029441 learned_addr learned my addr 172.21.15.94:0/1107029441 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.229+0000 7f47e6539640 1 -- 172.21.15.94:0/1107029441 >> v1:172.21.15.94:6793/0 conn(0x7f47e01370a0 legacy=0x7f47e013ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.229+0000 7f47e5537640 1 -- 172.21.15.94:0/1107029441 >> v1:172.21.15.94:6789/0 conn(0x7f47e0136cc0 legacy=0x7f47e013b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849540955 0 0) 0x7f47e006c8d0 con 0x7f47e01370a0 2024-09-16T10:30:03.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient(hunting): my global_id is 4499 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e5d38640 1 -- 172.21.15.94:0/1107029441 >> v1:172.21.15.152:6789/0 conn(0x7f47e01368e0 legacy=0x7f47e00a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47cc003520 con 0x7f47e01370a0 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555277077 0 0) 0x7f47e0061ae0 con 0x7f47e0136cc0 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient(hunting): my global_id is 4509 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e006c8d0 con 0x7f47e0136cc0 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4034680397 0 0) 0x7f47cc003520 con 0x7f47e01370a0 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 >> v1:172.21.15.152:6789/0 conn(0x7f47e01368e0 legacy=0x7f47e00a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 >> v1:172.21.15.94:6789/0 conn(0x7f47e0136cc0 legacy=0x7f47e013b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 monclient: found mon.noname-e 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e00a0680 con 0x7f47e01370a0 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f47d8001de0 con 0x7f47e01370a0 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e7f9c640 5 monclient: authenticate success, global_id 4499 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e7f9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:03.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: dump: 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.238 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e7f9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47d80020f0 con 0x7f47e01370a0 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 1 -- 172.21.15.94:0/1107029441 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f47d8004ea0 con 0x7f47e01370a0 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e7f9c640 10 monclient: get_monmap_and_config success 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e7f9c640 4 set_mon_vals no callback set 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e4d36640 10 monclient: dump: 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.240 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e7f9c640 10 monclient: shutdown 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e7f9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.230+0000 7f47e7f9c640 1 -- 172.21.15.94:0/1107029441 >> v1:172.21.15.94:6793/0 conn(0x7f47e01370a0 legacy=0x7f47e013ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.231+0000 7f47e7f9c640 1 -- 172.21.15.94:0/1107029441 shutdown_connections 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.231+0000 7f47e7f9c640 1 -- 172.21.15.94:0/1107029441 >> 172.21.15.94:0/1107029441 conn(0x7f47e00a58d0 msgr2=0x7f47e00a5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.231+0000 7f47e7f9c640 1 -- 172.21.15.94:0/1107029441 shutdown_connections 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.231+0000 7f47e7f9c640 1 -- 172.21.15.94:0/1107029441 wait complete. 2024-09-16T10:30:03.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.231+0000 7f47e7f9c640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.231+0000 7f47e7f9c640 10 monclient: monmap: 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.242 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.231+0000 7f47e7f9c640 1 Processor -- start 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 1 -- start start 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 10 monclient: init 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 10 monclient: _add_conns ranks=[6,1,8,7,4,2,5,3,0] 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 10 monclient(hunting): picked mon.noname-d con 0x7f47e01368e0 addr v1:172.21.15.94:6792/0 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 10 monclient(hunting): picked mon.noname-f con 0x7f47e0136cc0 addr v1:172.21.15.152:6789/0 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 10 monclient(hunting): picked mon.noname-e con 0x7f47e01370a0 addr v1:172.21.15.94:6793/0 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e5d38640 1 -- >> v1:172.21.15.94:6792/0 conn(0x7f47e01368e0 legacy=0x7f47e0135790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6792/0 2024-09-16T10:30:03.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e00a87d0 con 0x7f47e0136cc0 2024-09-16T10:30:03.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e0121950 con 0x7f47e01368e0 2024-09-16T10:30:03.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e0120da0 con 0x7f47e01370a0 2024-09-16T10:30:03.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e7f9c640 10 monclient(hunting): authenticate will time out at 770.572509s 2024-09-16T10:30:03.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e6539640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f47e01370a0 0x7f47e01d30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:52080/0 (socket says 172.21.15.94:52080) 2024-09-16T10:30:03.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.232+0000 7f47e6539640 1 -- 172.21.15.94:0/755209427 learned_addr learned my addr 172.21.15.94:0/755209427 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47e6539640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.94:6793/0 conn(0x7f47e01370a0 legacy=0x7f47e01d30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029518847 0 0) 0x7f47e0120da0 con 0x7f47e01370a0 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 10 monclient(hunting): my global_id is 4508 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c4003520 con 0x7f47e01370a0 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1417788847 0 0) 0x7f47c4003520 con 0x7f47e01370a0 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47e0120da0 con 0x7f47e01370a0 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f47d8004ea0 con 0x7f47e01370a0 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.234+0000 7f47d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:30:03.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47e5537640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.152:6789/0 conn(0x7f47e0136cc0 legacy=0x7f47e01cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient(hunting): dump: 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.249 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2212944353 0 0) 0x7f47e0120da0 con 0x7f47e01370a0 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.94:6792/0 conn(0x7f47e01368e0 legacy=0x7f47e0135790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.152:6789/0 conn(0x7f47e0136cc0 legacy=0x7f47e01cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 1 monclient: found mon.e 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e00a8310 con 0x7f47e01370a0 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47e7f9c640 5 monclient: authenticate success, global_id 4508 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47e7f9c640 10 monclient: _renew_subs 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47e7f9c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.94:6793/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e00a29c0 con 0x7f47e01370a0 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47e7f9c640 10 monclient: _renew_subs 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47e7f9c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e0123150 con 0x7f47e01370a0 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.8 v1:172.21.15.94:6793/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47d80052b0 con 0x7f47e01370a0 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.8 v1:172.21.15.94:6793/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f47d8005760 con 0x7f47e01370a0 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d3fff640 4 set_mon_vals no callback set 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.235+0000 7f47d27fc640 10 monclient: dump: 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.252 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.236+0000 7f47e7f9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.236+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f47c8006580 con 0x7f47c8002820 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.236+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.8 v1:172.21.15.94:6793/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f47d8005980 con 0x7f47e01370a0 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.236+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.8 v1:172.21.15.94:6793/0 8 ==== osd_map(20..20 src has 1..20) v4 ==== 3898+0+0 (unknown 541210984 0 0) 0x7f47d80465d0 con 0x7f47e01370a0 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.236+0000 7f47e5537640 10 monclient: get_auth_request con 0x7f47c40390c0 auth_method 0 2024-09-16T10:30:03.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47e5d38640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.152:6789/0 conn(0x7f47c8002820 legacy=0x7f47c8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212347507 0 0) 0x7f47c8006580 con 0x7f47c8002820 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 10 monclient(hunting): my global_id is 4465 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e0120da0 con 0x7f47c8002820 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3406068453 0 0) 0x7f47e0120da0 con 0x7f47c8002820 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47c8006580 con 0x7f47c8002820 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f47dc001b60 con 0x7f47c8002820 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:30:03.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.238+0000 7f47d27fc640 10 monclient: dump: 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.239+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3071998281 0 0) 0x7f47c8006580 con 0x7f47c8002820 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.239+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.152:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47c800a0d0 con 0x7f47c8002820 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.239+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f47dc0057c0 con 0x7f47c8002820 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.239+0000 7f47d27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.239+0000 7f47d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.239+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.152:6789/0 conn(0x7f47c8002820 legacy=0x7f47c8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.239+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.152:6789/0 conn(0x7f47c8002820 legacy=0x7f47c8004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.245+0000 7f47e7f9c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:30:03.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.245+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f47c8009d90 con 0x7f47c80090b0 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.248+0000 7f47e5d38640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.152:6789/0 conn(0x7f47c80090b0 legacy=0x7f47c8009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.248+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345435347 0 0) 0x7f47c8009d90 con 0x7f47c80090b0 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.248+0000 7f47d27fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.248+0000 7f47d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.248+0000 7f47d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.248+0000 7f47d27fc640 10 monclient(hunting): my global_id is 4474 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.248+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c8006580 con 0x7f47c80090b0 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 709118611 0 0) 0x7f47c8006580 con 0x7f47c80090b0 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47c8009d90 con 0x7f47c80090b0 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f47dc003ec0 con 0x7f47c80090b0 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 10 monclient: dump: 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.259 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3346983897 0 0) 0x7f47c8009d90 con 0x7f47c80090b0 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 --> v1:172.21.15.152:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f47c800c280 con 0x7f47c80090b0 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 <== mon.1 v1:172.21.15.152:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7441 (unknown 0 0 2418887928) 0x7f47dc0043a0 con 0x7f47c80090b0 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.152:6789/0 conn(0x7f47c80090b0 legacy=0x7f47c8009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.249+0000 7f47d27fc640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.152:6789/0 conn(0x7f47c80090b0 legacy=0x7f47c8009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 14, 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:30:03.261 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 1, 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.262 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.263 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:30:03.264 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.265 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:30:03.266 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.267 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.268 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:30:03.269 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.270 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.272 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.274 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.275 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.277 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.278 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.279 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.280 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:03.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.280 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.280 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:30:03.280 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:30:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.152:6800/2686360781 conn(0x7f47c40390c0 legacy=0x7f47c403b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e7f9c640 10 monclient: shutdown 2024-09-16T10:30:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e7f9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 >> v1:172.21.15.94:6793/0 conn(0x7f47e01370a0 legacy=0x7f47e01d30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e6539640 1 -- 172.21.15.94:0/755209427 reap_dead start 2024-09-16T10:30:03.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e6539640 1 -- 172.21.15.94:0/755209427 reap_dead start 2024-09-16T10:30:03.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 shutdown_connections 2024-09-16T10:30:03.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 >> 172.21.15.94:0/755209427 conn(0x7f47e00a58d0 msgr2=0x7f47e01be2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 shutdown_connections 2024-09-16T10:30:03.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.250+0000 7f47e7f9c640 1 -- 172.21.15.94:0/755209427 wait complete. 2024-09-16T10:30:03.282 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-16T10:30:03.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-09-16T10:30:03.282 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-16T10:30:03.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.390+0000 7f2e2a25a640 10 monclient: get_monmap_and_config 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.390+0000 7f2e2a25a640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.390+0000 7f2e2a25a640 10 monclient: monmap: 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:30:03.391663+0000 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:30:03.391663+0000 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.394 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.394 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.394 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.394 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.394 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.394 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.391+0000 7f2e2a25a640 1 Processor -- start 2024-09-16T10:30:03.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.391+0000 7f2e2a25a640 1 -- start start 2024-09-16T10:30:03.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.391+0000 7f2e2a25a640 10 monclient: init 2024-09-16T10:30:03.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 10 monclient: _add_conns ranks=[2,4,8,7,3,6,1,0,5] 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 10 monclient(hunting): picked mon.noname-b con 0x7f2e240a38d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 10 monclient(hunting): picked mon.noname-c con 0x7f2e2406c670 addr v1:172.21.15.94:6791/0 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 10 monclient(hunting): picked mon.noname-e con 0x7f2e240a43c0 addr v1:172.21.15.94:6793/0 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e24061ae0 con 0x7f2e240a38d0 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240a04d0 con 0x7f2e2406c670 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240a3250 con 0x7f2e240a43c0 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e2a25a640 10 monclient(hunting): authenticate will time out at 770.732459s 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e22ffd640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f2e2406c670 0x7f2e2413bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:45774/0 (socket says 172.21.15.94:45774) 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e237fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f2e240a38d0 0x7f2e240a3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:54296/0 (socket says 172.21.15.94:54296) 2024-09-16T10:30:03.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.392+0000 7f2e22ffd640 1 -- 172.21.15.94:0/4065501284 learned_addr learned my addr 172.21.15.94:0/4065501284 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e237fe640 1 -- 172.21.15.94:0/4065501284 >> v1:172.21.15.94:6790/0 conn(0x7f2e240a38d0 legacy=0x7f2e240a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e22ffd640 1 -- 172.21.15.94:0/4065501284 >> v1:172.21.15.94:6791/0 conn(0x7f2e2406c670 legacy=0x7f2e2413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e23fff640 1 -- 172.21.15.94:0/4065501284 >> v1:172.21.15.94:6793/0 conn(0x7f2e240a43c0 legacy=0x7f2e2413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034833284 0 0) 0x7f2e240a04d0 con 0x7f2e2406c670 2024-09-16T10:30:03.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): my global_id is 4549 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e0c003520 con 0x7f2e2406c670 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186077894 0 0) 0x7f2e24061ae0 con 0x7f2e240a38d0 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): my global_id is 4511 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e240a04d0 con 0x7f2e240a38d0 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266278799 0 0) 0x7f2e240a3250 con 0x7f2e240a43c0 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 10 monclient(hunting): my global_id is 4517 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.394+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e24061ae0 con 0x7f2e240a43c0 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 24728981 0 0) 0x7f2e0c003520 con 0x7f2e2406c670 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 >> v1:172.21.15.94:6793/0 conn(0x7f2e240a43c0 legacy=0x7f2e2413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 >> v1:172.21.15.94:6790/0 conn(0x7f2e240a38d0 legacy=0x7f2e240a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 1 monclient: found mon.noname-c 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:30:03.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e240a0a00 con 0x7f2e2406c670 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2e18001990 con 0x7f2e2406c670 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: dump: 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.403 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.404 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e2a25a640 5 monclient: authenticate success, global_id 4549 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e2a25a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e18001ca0 con 0x7f2e2406c670 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 1 -- 172.21.15.94:0/4065501284 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2e18004be0 con 0x7f2e2406c670 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e2a25a640 10 monclient: get_monmap_and_config success 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e2a25a640 4 set_mon_vals no callback set 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e227fc640 10 monclient: dump: 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e2a25a640 10 monclient: shutdown 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e2a25a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.395+0000 7f2e2a25a640 1 -- 172.21.15.94:0/4065501284 >> v1:172.21.15.94:6791/0 conn(0x7f2e2406c670 legacy=0x7f2e2413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.396+0000 7f2e2a25a640 1 -- 172.21.15.94:0/4065501284 shutdown_connections 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.396+0000 7f2e2a25a640 1 -- 172.21.15.94:0/4065501284 >> 172.21.15.94:0/4065501284 conn(0x7f2e240a59d0 msgr2=0x7f2e240a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.396+0000 7f2e2a25a640 1 -- 172.21.15.94:0/4065501284 shutdown_connections 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.396+0000 7f2e2a25a640 1 -- 172.21.15.94:0/4065501284 wait complete. 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.396+0000 7f2e2a25a640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.396+0000 7f2e2a25a640 10 monclient: monmap: 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:30:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.408 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.396+0000 7f2e2a25a640 1 Processor -- start 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 1 -- start start 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 10 monclient: init 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 10 monclient: _add_conns ranks=[4,2,1,0,7,3,5,8,6] 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 10 monclient(hunting): picked mon.noname-c con 0x7f2e2406c670 addr v1:172.21.15.94:6791/0 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 10 monclient(hunting): picked mon.noname-b con 0x7f2e240a38d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 10 monclient(hunting): picked mon.noname-f con 0x7f2e240a43c0 addr v1:172.21.15.152:6789/0 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e24127360 con 0x7f2e240a43c0 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2409ecc0 con 0x7f2e240a38d0 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2409eff0 con 0x7f2e2406c670 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e2a25a640 10 monclient(hunting): authenticate will time out at 770.737806s 2024-09-16T10:30:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e237fe640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f2e2406c670 0x7f2e24136970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:45780/0 (socket says 172.21.15.94:45780) 2024-09-16T10:30:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.397+0000 7f2e237fe640 1 -- 172.21.15.94:0/1103543364 learned_addr learned my addr 172.21.15.94:0/1103543364 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.400+0000 7f2e237fe640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.94:6791/0 conn(0x7f2e2406c670 legacy=0x7f2e24136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.400+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 DispatchQueue::entry inject delay of 0.002000 2024-09-16T10:30:03.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.400+0000 7f2e22ffd640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.94:6790/0 conn(0x7f2e240a38d0 legacy=0x7f2e241cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.400+0000 7f2e23fff640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.152:6789/0 conn(0x7f2e240a43c0 legacy=0x7f2e241d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091847286 0 0) 0x7f2e2409eff0 con 0x7f2e2406c670 2024-09-16T10:30:03.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): my global_id is 4558 2024-09-16T10:30:03.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e10003520 con 0x7f2e2406c670 2024-09-16T10:30:03.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460456887 0 0) 0x7f2e2409ecc0 con 0x7f2e240a38d0 2024-09-16T10:30:03.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): my global_id is 4520 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e2409eff0 con 0x7f2e240a38d0 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385838287 0 0) 0x7f2e24127360 con 0x7f2e240a43c0 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): my global_id is 4483 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e2409ecc0 con 0x7f2e240a43c0 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 634268227 0 0) 0x7f2e10003520 con 0x7f2e2406c670 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e24127360 con 0x7f2e2406c670 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2e14001920 con 0x7f2e2406c670 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): dump: 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.419 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.419 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.419 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.419 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.419 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.419 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.419 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:30:03.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1205332951 0 0) 0x7f2e2409eff0 con 0x7f2e240a38d0 2024-09-16T10:30:03.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e10003520 con 0x7f2e240a38d0 2024-09-16T10:30:03.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2e18004b20 con 0x7f2e240a38d0 2024-09-16T10:30:03.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:30:03.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): dump: 2024-09-16T10:30:03.420 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.420 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.420 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.421 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.421 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.421 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.421 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.421 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.421 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.421 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.421 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 322133809 0 0) 0x7f2e2409ecc0 con 0x7f2e240a43c0 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e2409eff0 con 0x7f2e240a43c0 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2e1c003cd0 con 0x7f2e240a43c0 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient(hunting): dump: 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.423 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 773173601 0 0) 0x7f2e24127360 con 0x7f2e2406c670 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.94:6790/0 conn(0x7f2e240a38d0 legacy=0x7f2e241cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.152:6789/0 conn(0x7f2e240a43c0 legacy=0x7f2e241d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 monclient: found mon.c 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2406b780 con 0x7f2e2406c670 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e2a25a640 5 monclient: authenticate success, global_id 4558 2024-09-16T10:30:03.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e2a25a640 10 monclient: _renew_subs 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e2a25a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e24127080 con 0x7f2e2406c670 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e2a25a640 10 monclient: _renew_subs 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e2a25a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e24124f90 con 0x7f2e2406c670 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e14004dd0 con 0x7f2e2406c670 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2e14005280 con 0x7f2e2406c670 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e21ffb640 4 set_mon_vals no callback set 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.402+0000 7f2e0bfff640 10 monclient: dump: 2024-09-16T10:30:03.425 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.426 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.403+0000 7f2e2a25a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.403+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2e04006580 con 0x7f2e04002820 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.403+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f2e140102e0 con 0x7f2e2406c670 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.404+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(21..21 src has 1..21) v4 ==== 3898+0+0 (unknown 3376581462 0 0) 0x7f2e14046220 con 0x7f2e2406c670 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.404+0000 7f2e23fff640 10 monclient: get_auth_request con 0x7f2e10039dc0 auth_method 0 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e22ffd640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.152:6790/0 conn(0x7f2e04002820 legacy=0x7f2e04004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818122172 0 0) 0x7f2e04006580 con 0x7f2e04002820 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 10 monclient(hunting): my global_id is 4683 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e24127360 con 0x7f2e04002820 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 310927290 0 0) 0x7f2e24127360 con 0x7f2e04002820 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e04006580 con 0x7f2e04002820 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2e180014b0 con 0x7f2e04002820 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 10 monclient: dump: 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4005558612 0 0) 0x7f2e04006580 con 0x7f2e04002820 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.406+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e0400a0d0 con 0x7f2e04002820 2024-09-16T10:30:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.407+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2e18005490 con 0x7f2e04002820 2024-09-16T10:30:03.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.407+0000 7f2e0bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.407+0000 7f2e0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:30:03.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.407+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.152:6790/0 conn(0x7f2e04002820 legacy=0x7f2e04004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.407+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.152:6790/0 conn(0x7f2e04002820 legacy=0x7f2e04004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.413+0000 7f2e2a25a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:30:03.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.413+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2e04009d90 con 0x7f2e040090b0 2024-09-16T10:30:03.431 INFO:teuthology.orchestra.run.smithi152.stdout:******************************* 2024-09-16T10:30:03.431 INFO:teuthology.orchestra.run.smithi152.stdout:process_3_[34865]: starting. 2024-09-16T10:30:03.431 INFO:teuthology.orchestra.run.smithi152.stdout:process_3_[34865]: creating pool ceph_test_rados_delete_pools_parallel.smithi152-34830 2024-09-16T10:30:03.431 INFO:teuthology.orchestra.run.smithi152.stdout:process_3_[34865]: created object 0... 2024-09-16T10:30:03.431 INFO:teuthology.orchestra.run.smithi152.stdout:process_3_[34865]: created object 25... 2024-09-16T10:30:03.431 INFO:teuthology.orchestra.run.smithi152.stdout:process_3_[34865]: created object 49... 2024-09-16T10:30:03.431 INFO:teuthology.orchestra.run.smithi152.stdout:process_3_[34865]: finishing. 2024-09-16T10:30:03.431 INFO:teuthology.orchestra.run.smithi152.stdout:process_3_[34865]: shutting down. 2024-09-16T10:30:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.415+0000 7f2e22ffd640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.152:6790/0 conn(0x7f2e040090b0 legacy=0x7f2e04009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.415+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202996253 0 0) 0x7f2e04009d90 con 0x7f2e040090b0 2024-09-16T10:30:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.415+0000 7f2e0bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.415+0000 7f2e0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.415+0000 7f2e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.415+0000 7f2e0bfff640 10 monclient(hunting): my global_id is 4692 2024-09-16T10:30:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e04006580 con 0x7f2e040090b0 2024-09-16T10:30:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 546196274 0 0) 0x7f2e04006580 con 0x7f2e040090b0 2024-09-16T10:30:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e04009d90 con 0x7f2e040090b0 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f2e18005cc0 con 0x7f2e040090b0 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 10 monclient: dump: 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.435 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3086446193 0 0) 0x7f2e04009d90 con 0x7f2e040090b0 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 --> v1:172.21.15.152:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2e0400c280 con 0x7f2e040090b0 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 <== mon.3 v1:172.21.15.152:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7607 (unknown 0 0 4227926171) 0x7f2e180060d0 con 0x7f2e040090b0 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.152:6790/0 conn(0x7f2e040090b0 legacy=0x7f2e04009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.416+0000 7f2e0bfff640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.152:6790/0 conn(0x7f2e040090b0 legacy=0x7f2e04009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:30:03.436 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 14, 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.437 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 1, 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.438 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.439 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:30:03.440 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.441 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:30:03.442 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.443 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:30:03.446 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.447 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:30:03.449 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.450 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.451 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.452 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.454 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.455 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.455 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.455 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.455 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.455 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.455 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:30:03.455 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:30:03.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.456 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:30:03.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.457 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.457 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.457 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:30:03.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.457 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.457 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:30:03.457 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:30:03.458 INFO:teuthology.orchestra.run.smithi152.stdout:******************************* 2024-09-16T10:30:03.459 INFO:teuthology.orchestra.run.smithi152.stdout:process_5_[34867]: starting. 2024-09-16T10:30:03.459 INFO:teuthology.orchestra.run.smithi152.stdout:process_5_[34867]: listing objects. 2024-09-16T10:30:03.459 INFO:teuthology.orchestra.run.smithi152.stdout:process_5_[34867]: listed object 0... 2024-09-16T10:30:03.459 INFO:teuthology.orchestra.run.smithi152.stdout:process_5_[34867]: listed object 25... 2024-09-16T10:30:03.459 INFO:teuthology.orchestra.run.smithi152.stdout:process_5_[34867]: saw 50 objects 2024-09-16T10:30:03.459 INFO:teuthology.orchestra.run.smithi152.stdout:process_5_[34867]: shutting down. 2024-09-16T10:30:03.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.152:6800/2686360781 conn(0x7f2e10039dc0 legacy=0x7f2e1003c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e2a25a640 10 monclient: shutdown 2024-09-16T10:30:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e2a25a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 >> v1:172.21.15.94:6791/0 conn(0x7f2e2406c670 legacy=0x7f2e24136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e23fff640 1 -- 172.21.15.94:0/1103543364 reap_dead start 2024-09-16T10:30:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e23fff640 1 -- 172.21.15.94:0/1103543364 reap_dead start 2024-09-16T10:30:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 shutdown_connections 2024-09-16T10:30:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 >> 172.21.15.94:0/1103543364 conn(0x7f2e240a59d0 msgr2=0x7f2e2409ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 shutdown_connections 2024-09-16T10:30:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.417+0000 7f2e2a25a640 1 -- 172.21.15.94:0/1103543364 wait complete. 2024-09-16T10:30:03.461 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-16T10:30:03.461 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-09-16T10:30:03.461 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-16T10:30:03.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.557+0000 7f6dd01ae640 10 monclient: get_monmap_and_config 2024-09-16T10:30:03.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.557+0000 7f6dd01ae640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.557+0000 7f6dd01ae640 10 monclient: monmap: 2024-09-16T10:30:03.559 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.559 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.559 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:30:03.558120+0000 2024-09-16T10:30:03.559 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:30:03.558120+0000 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.560 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.557+0000 7f6dd01ae640 1 Processor -- start 2024-09-16T10:30:03.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 1 -- start start 2024-09-16T10:30:03.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 10 monclient: init 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 10 monclient: _add_conns ranks=[8,1,7,2,6,4,3,5,0] 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 10 monclient(hunting): picked mon.noname-e con 0x7f6dc80a3860 addr v1:172.21.15.94:6793/0 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 10 monclient(hunting): picked mon.noname-f con 0x7f6dc806c1d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 10 monclient(hunting): picked mon.noname-i con 0x7f6dc80a4350 addr v1:172.21.15.152:6792/0 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc8061ae0 con 0x7f6dc806c1d0 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc80a0600 con 0x7f6dc80a4350 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc80a0b30 con 0x7f6dc80a3860 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dd01ae640 10 monclient(hunting): authenticate will time out at 770.898580s 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dcdf4a640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f6dc80a3860 0x7f6dc80a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:52086/0 (socket says 172.21.15.94:52086) 2024-09-16T10:30:03.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.558+0000 7f6dcdf4a640 1 -- 172.21.15.94:0/4094484174 learned_addr learned my addr 172.21.15.94:0/4094484174 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.560+0000 7f6dcdf4a640 1 -- 172.21.15.94:0/4094484174 >> v1:172.21.15.94:6793/0 conn(0x7f6dc80a3860 legacy=0x7f6dc80a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.560+0000 7f6dccf48640 1 -- 172.21.15.94:0/4094484174 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798922893 0 0) 0x7f6dc80a0b30 con 0x7f6dc80a3860 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.560+0000 7f6dccf48640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.560+0000 7f6dccf48640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.560+0000 7f6dccf48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.560+0000 7f6dccf48640 10 monclient(hunting): my global_id is 4526 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 1 -- 172.21.15.94:0/4094484174 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db4003520 con 0x7f6dc80a3860 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dce74b640 1 -- 172.21.15.94:0/4094484174 >> v1:172.21.15.152:6792/0 conn(0x7f6dc80a4350 legacy=0x7f6dc813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dcd749640 1 -- 172.21.15.94:0/4094484174 >> v1:172.21.15.152:6789/0 conn(0x7f6dc806c1d0 legacy=0x7f6dc813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 1 -- 172.21.15.94:0/4094484174 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2584568569 0 0) 0x7f6db4003520 con 0x7f6dc80a3860 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 1 -- 172.21.15.94:0/4094484174 >> v1:172.21.15.152:6792/0 conn(0x7f6dc80a4350 legacy=0x7f6dc813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 1 -- 172.21.15.94:0/4094484174 >> v1:172.21.15.152:6789/0 conn(0x7f6dc806c1d0 legacy=0x7f6dc813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 1 monclient: found mon.noname-e 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 1 -- 172.21.15.94:0/4094484174 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc80a0e10 con 0x7f6dc80a3860 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 1 -- 172.21.15.94:0/4094484174 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6dc4001b20 con 0x7f6dc80a3860 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: dump: 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.572 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 5 monclient: authenticate success, global_id 4526 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 1 -- 172.21.15.94:0/4094484174 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dc40048a0 con 0x7f6dc80a3860 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 1 -- 172.21.15.94:0/4094484174 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6dc4004d50 con 0x7f6dc80a3860 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 10 monclient: get_monmap_and_config success 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 4 set_mon_vals no callback set 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dccf48640 10 monclient: dump: 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.574 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.577 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.577 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 10 monclient: shutdown 2024-09-16T10:30:03.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 1 -- 172.21.15.94:0/4094484174 >> v1:172.21.15.94:6793/0 conn(0x7f6dc80a3860 legacy=0x7f6dc80a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 1 -- 172.21.15.94:0/4094484174 shutdown_connections 2024-09-16T10:30:03.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 1 -- 172.21.15.94:0/4094484174 >> 172.21.15.94:0/4094484174 conn(0x7f6dc80a5960 msgr2=0x7f6dc80a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 1 -- 172.21.15.94:0/4094484174 shutdown_connections 2024-09-16T10:30:03.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.561+0000 7f6dd01ae640 1 -- 172.21.15.94:0/4094484174 wait complete. 2024-09-16T10:30:03.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.562+0000 7f6dd01ae640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.562+0000 7f6dd01ae640 10 monclient: monmap: 2024-09-16T10:30:03.578 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:30:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:30:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:30:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.581 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.562+0000 7f6dd01ae640 1 Processor -- start 2024-09-16T10:30:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.562+0000 7f6dd01ae640 1 -- start start 2024-09-16T10:30:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.562+0000 7f6dd01ae640 10 monclient: init 2024-09-16T10:30:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 10 monclient: _add_conns ranks=[3,7,6,1,2,8,5,4,0] 2024-09-16T10:30:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 10 monclient(hunting): picked mon.noname-g con 0x7f6dc806c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:30:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 10 monclient(hunting): picked mon.noname-i con 0x7f6dc80a3860 addr v1:172.21.15.152:6792/0 2024-09-16T10:30:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 10 monclient(hunting): picked mon.noname-d con 0x7f6dc80a4350 addr v1:172.21.15.94:6792/0 2024-09-16T10:30:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc809f100 con 0x7f6dc806c1d0 2024-09-16T10:30:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc81abf30 con 0x7f6dc80a4350 2024-09-16T10:30:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc81ac210 con 0x7f6dc80a3860 2024-09-16T10:30:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dd01ae640 10 monclient(hunting): authenticate will time out at 770.903301s 2024-09-16T10:30:03.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dce74b640 1 -- >> v1:172.21.15.94:6792/0 conn(0x7f6dc80a4350 legacy=0x7f6dc81d3490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6792/0 2024-09-16T10:30:03.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dcdf4a640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7f6dc806c1d0 0x7f6dc81be850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.94:49390/0 (socket says 172.21.15.94:49390) 2024-09-16T10:30:03.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.563+0000 7f6dcdf4a640 1 -- 172.21.15.94:0/3163011300 learned_addr learned my addr 172.21.15.94:0/3163011300 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dcd749640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6792/0 conn(0x7f6dc80a3860 legacy=0x7f6dc81cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dcdf4a640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6790/0 conn(0x7f6dc806c1d0 legacy=0x7f6dc81be850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693610874 0 0) 0x7f6dc809f100 con 0x7f6dc806c1d0 2024-09-16T10:30:03.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 10 monclient(hunting): my global_id is 4701 2024-09-16T10:30:03.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dac003520 con 0x7f6dc806c1d0 2024-09-16T10:30:03.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907352853 0 0) 0x7f6dc81ac210 con 0x7f6dc80a3860 2024-09-16T10:30:03.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 10 monclient(hunting): my global_id is 4543 2024-09-16T10:30:03.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc809f100 con 0x7f6dc80a3860 2024-09-16T10:30:03.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.565+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2607024994 0 0) 0x7f6dac003520 con 0x7f6dc806c1d0 2024-09-16T10:30:03.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dc81ac210 con 0x7f6dc806c1d0 2024-09-16T10:30:03.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6dc4001ae0 con 0x7f6dc806c1d0 2024-09-16T10:30:03.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:30:03.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient(hunting): dump: 2024-09-16T10:30:03.586 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.586 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.586 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.587 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.587 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.587 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.587 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.587 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.587 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.588 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.588 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.588 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.588 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.588 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.588 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.589 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:30:03.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 846393877 0 0) 0x7f6dc809f100 con 0x7f6dc80a3860 2024-09-16T10:30:03.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dac003520 con 0x7f6dc80a3860 2024-09-16T10:30:03.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6dbc001c60 con 0x7f6dc80a3860 2024-09-16T10:30:03.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:30:03.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient(hunting): dump: 2024-09-16T10:30:03.590 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.590 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.590 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.590 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.590 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.591 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.591 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.591 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.591 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.591 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.591 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.591 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.591 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.592 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.592 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.592 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 896921121 0 0) 0x7f6dc81ac210 con 0x7f6dc806c1d0 2024-09-16T10:30:03.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6792/0 conn(0x7f6dc80a3860 legacy=0x7f6dc81cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.94:6792/0 conn(0x7f6dc80a4350 legacy=0x7f6dc81d3490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:30:03.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 monclient: found mon.g 2024-09-16T10:30:03.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:03.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc809f650 con 0x7f6dc806c1d0 2024-09-16T10:30:03.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dd01ae640 5 monclient: authenticate success, global_id 4701 2024-09-16T10:30:03.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dd01ae640 10 monclient: _renew_subs 2024-09-16T10:30:03.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dd01ae640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:03.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc81aa790 con 0x7f6dc806c1d0 2024-09-16T10:30:03.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dd01ae640 10 monclient: _renew_subs 2024-09-16T10:30:03.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dd01ae640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:03.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc81bef60 con 0x7f6dc806c1d0 2024-09-16T10:30:03.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dc4001320 con 0x7f6dc806c1d0 2024-09-16T10:30:03.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dbbfff640 4 set_mon_vals no callback set 2024-09-16T10:30:03.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6dc4004d70 con 0x7f6dc806c1d0 2024-09-16T10:30:03.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:30:03.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.566+0000 7f6dba7fc640 10 monclient: dump: 2024-09-16T10:30:03.596 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.596 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.596 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.597 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.597 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.597 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.597 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.597 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.597 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.598 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.598 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.598 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.598 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.598 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.598 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.598 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.567+0000 7f6dd01ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.567+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6db0006580 con 0x7f6db0002820 2024-09-16T10:30:03.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.567+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f6dc4010570 con 0x7f6dc806c1d0 2024-09-16T10:30:03.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.567+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(21..21 src has 1..21) v4 ==== 3898+0+0 (unknown 3376581462 0 0) 0x7f6dc4046380 con 0x7f6dc806c1d0 2024-09-16T10:30:03.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.567+0000 7f6dce74b640 10 monclient: get_auth_request con 0x7f6dac039780 auth_method 0 2024-09-16T10:30:03.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dcd749640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6791/0 conn(0x7f6db0002820 legacy=0x7f6db0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203086357 0 0) 0x7f6db0006580 con 0x7f6db0002820 2024-09-16T10:30:03.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 10 monclient(hunting): my global_id is 4568 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc81ac210 con 0x7f6db0002820 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2212926913 0 0) 0x7f6dc81ac210 con 0x7f6db0002820 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6db0006580 con 0x7f6db0002820 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6dbc003cf0 con 0x7f6db0002820 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 10 monclient: dump: 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2529727649 0 0) 0x7f6db0006580 con 0x7f6db0002820 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.569+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6db000a0d0 con 0x7f6db0002820 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.570+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6dbc0041f0 con 0x7f6db0002820 2024-09-16T10:30:03.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.570+0000 7f6dba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.570+0000 7f6dba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.570+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6791/0 conn(0x7f6db0002820 legacy=0x7f6db0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.570+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6791/0 conn(0x7f6db0002820 legacy=0x7f6db0004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.576+0000 7f6dd01ae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.576+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6db0009d90 con 0x7f6db00090b0 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.578+0000 7f6dcd749640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6791/0 conn(0x7f6db00090b0 legacy=0x7f6db0009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177179205 0 0) 0x7f6db0009d90 con 0x7f6db00090b0 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 10 monclient(hunting): my global_id is 4577 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db0006580 con 0x7f6db00090b0 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1809907058 0 0) 0x7f6db0006580 con 0x7f6db00090b0 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6db0009d90 con 0x7f6db00090b0 2024-09-16T10:30:03.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6dbc005b00 con 0x7f6db00090b0 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 10 monclient: dump: 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.605 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2108471049 0 0) 0x7f6db0009d90 con 0x7f6db00090b0 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 --> v1:172.21.15.152:6791/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6db000c280 con 0x7f6db00090b0 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 <== mon.5 v1:172.21.15.152:6791/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7607 (unknown 0 0 1319728693) 0x7f6dbc006010 con 0x7f6db00090b0 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6791/0 conn(0x7f6db00090b0 legacy=0x7f6db0009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.579+0000 7f6dba7fc640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6791/0 conn(0x7f6db00090b0 legacy=0x7f6db0009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.607 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:30:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:30:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:30:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:30:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 14, 2024-09-16T10:30:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:30:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:30:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:30:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:30:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:30:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:30:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:30:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:30:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:30:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 1, 2024-09-16T10:30:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:30:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:30:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:30:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:30:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:30:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:30:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:30:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:30:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:30:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:30:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:30:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:30:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:30:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:30:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:30:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:30:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:30:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:30:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:30:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:30:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:30:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:30:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:30:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:30:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:30:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:30:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:30:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:30:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:30:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:30:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:30:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:30:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:30:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:30:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:30:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:30:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:30:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:30:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:30:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:30:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:30:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:30:03.640 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6800/2686360781 conn(0x7f6dac039780 legacy=0x7f6dac03bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dd01ae640 10 monclient: shutdown 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dd01ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 >> v1:172.21.15.152:6790/0 conn(0x7f6dc806c1d0 legacy=0x7f6dc81be850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dce74b640 1 -- 172.21.15.94:0/3163011300 reap_dead start 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dce74b640 1 -- 172.21.15.94:0/3163011300 reap_dead start 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 shutdown_connections 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 >> 172.21.15.94:0/3163011300 conn(0x7f6dc80a5960 msgr2=0x7f6dc809ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 shutdown_connections 2024-09-16T10:30:03.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.580+0000 7f6dd01ae640 1 -- 172.21.15.94:0/3163011300 wait complete. 2024-09-16T10:30:03.643 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-16T10:30:03.643 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-09-16T10:30:03.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-16T10:30:03.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.739+0000 7f347389c640 10 monclient: get_monmap_and_config 2024-09-16T10:30:03.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.739+0000 7f347389c640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.739+0000 7f347389c640 10 monclient: monmap: 2024-09-16T10:30:03.741 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.741 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:30:03.740431+0000 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:30:03.740431+0000 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.742 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.743 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.743 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.743 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.743 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 1 Processor -- start 2024-09-16T10:30:03.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 1 -- start start 2024-09-16T10:30:03.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 10 monclient: init 2024-09-16T10:30:03.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 10 monclient: _add_conns ranks=[1,0,6,2,5,3,7,4,8] 2024-09-16T10:30:03.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 10 monclient(hunting): picked mon.noname-f con 0x7f346c0a2620 addr v1:172.21.15.152:6789/0 2024-09-16T10:30:03.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 10 monclient(hunting): picked mon.noname-a con 0x7f346c06ec00 addr v1:172.21.15.94:6789/0 2024-09-16T10:30:03.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 10 monclient(hunting): picked mon.noname-d con 0x7f346c071420 addr v1:172.21.15.94:6792/0 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c061ae0 con 0x7f346c06ec00 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c070a40 con 0x7f346c0a2620 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c06f100 con 0x7f346c071420 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f347389c640 10 monclient(hunting): authenticate will time out at 771.080991s 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.740+0000 7f3471e39640 1 -- >> v1:172.21.15.94:6792/0 conn(0x7f346c071420 legacy=0x7f346c13cd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.94:6792/0 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.741+0000 7f3470e37640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f346c06ec00 0x7f346c139610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42118/0 (socket says 172.21.15.94:42118) 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.741+0000 7f3470e37640 1 -- 172.21.15.94:0/2478166619 learned_addr learned my addr 172.21.15.94:0/2478166619 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3470e37640 1 -- 172.21.15.94:0/2478166619 >> v1:172.21.15.94:6789/0 conn(0x7f346c06ec00 legacy=0x7f346c139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 -- 172.21.15.94:0/2478166619 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518664723 0 0) 0x7f346c061ae0 con 0x7f346c06ec00 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient(hunting): my global_id is 4518 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 -- 172.21.15.94:0/2478166619 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3458003520 con 0x7f346c06ec00 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3471638640 1 -- 172.21.15.94:0/2478166619 >> v1:172.21.15.152:6789/0 conn(0x7f346c0a2620 legacy=0x7f346c0a2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3471638640 1 -- 172.21.15.94:0/2478166619 >> v1:172.21.15.152:6789/0 conn(0x7f346c0a2620 legacy=0x7f346c0a2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3471638640 1 -- 172.21.15.94:0/2478166619 >> v1:172.21.15.152:6789/0 conn(0x7f346c0a2620 legacy=0x7f346c0a2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3471638640 1 --1- 172.21.15.94:0/2478166619 >> v1:172.21.15.152:6789/0 conn(0x7f346c0a2620 0x7f346c0a2a00 :-1 s=OPENED pgs=92 cs=1 l=1).handle_message read tag failed 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3471638640 1 --1- 172.21.15.94:0/2478166619 >> v1:172.21.15.152:6789/0 conn(0x7f346c0a2620 0x7f346c0a2a00 :-1 s=OPENED pgs=92 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.152:6789/0 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 -- 172.21.15.94:0/2478166619 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103606936 0 0) 0x7f3458003520 con 0x7f346c06ec00 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 -- 172.21.15.94:0/2478166619 >> v1:172.21.15.94:6792/0 conn(0x7f346c071420 legacy=0x7f346c13cd40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 -- 172.21.15.94:0/2478166619 >> v1:172.21.15.152:6789/0 conn(0x7f346c0a2620 legacy=0x7f346c0a2a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 monclient: found mon.noname-a 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 -- 172.21.15.94:0/2478166619 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c0701a0 con 0x7f346c06ec00 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f347389c640 5 monclient: authenticate success, global_id 4518 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f347389c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 -- 172.21.15.94:0/2478166619 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f345c001a60 con 0x7f346c06ec00 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: dump: 2024-09-16T10:30:03.749 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.750 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f347389c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 -- 172.21.15.94:0/2478166619 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f345c001d70 con 0x7f346c06ec00 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 1 -- 172.21.15.94:0/2478166619 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f345c004be0 con 0x7f346c06ec00 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f347389c640 10 monclient: get_monmap_and_config success 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f347389c640 4 set_mon_vals no callback set 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.743+0000 7f3463fff640 10 monclient: dump: 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.751 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.752 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.744+0000 7f347389c640 10 monclient: shutdown 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.744+0000 7f347389c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.744+0000 7f347389c640 1 -- 172.21.15.94:0/2478166619 >> v1:172.21.15.94:6789/0 conn(0x7f346c06ec00 legacy=0x7f346c139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.744+0000 7f347389c640 1 -- 172.21.15.94:0/2478166619 shutdown_connections 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.744+0000 7f347389c640 1 -- 172.21.15.94:0/2478166619 >> 172.21.15.94:0/2478166619 conn(0x7f346c0a7630 msgr2=0x7f346c0a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.744+0000 7f347389c640 1 -- 172.21.15.94:0/2478166619 shutdown_connections 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.744+0000 7f347389c640 1 -- 172.21.15.94:0/2478166619 wait complete. 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.744+0000 7f347389c640 10 monclient: build_initial_monmap 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.744+0000 7f347389c640 10 monclient: monmap: 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:30:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 1 Processor -- start 2024-09-16T10:30:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 1 -- start start 2024-09-16T10:30:03.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 10 monclient: init 2024-09-16T10:30:03.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 10 monclient: _reopen_session rank -1 2024-09-16T10:30:03.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 10 monclient: _add_conns ranks=[6,3,1,5,8,2,0,7,4] 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 10 monclient(hunting): picked mon.noname-d con 0x7f346c06ec00 addr v1:172.21.15.94:6792/0 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 10 monclient(hunting): picked mon.noname-g con 0x7f346c071420 addr v1:172.21.15.152:6790/0 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 10 monclient(hunting): picked mon.noname-f con 0x7f346c0a2620 addr v1:172.21.15.152:6789/0 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c0a0480 con 0x7f346c0a2620 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c09ff50 con 0x7f346c071420 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c06be20 con 0x7f346c06ec00 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 10 monclient(hunting): _renew_subs 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f3471638640 1 -- >> v1:172.21.15.94:6792/0 conn(0x7f346c06ec00 legacy=0x7f346c06b000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.94:6792/0 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f347389c640 10 monclient(hunting): authenticate will time out at 771.085625s 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f3471e39640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f346c0a2620 0x7f346c1d2be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.94:59266/0 (socket says 172.21.15.94:59266) 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f3471e39640 1 -- 172.21.15.94:0/3135706554 learned_addr learned my addr 172.21.15.94:0/3135706554 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:30:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.745+0000 7f3470e37640 1 --1- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6790/0 conn(0x7f346c071420 0x7f346c06b710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.94:49396/0 (socket says 172.21.15.94:49396) 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3470e37640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6790/0 conn(0x7f346c071420 legacy=0x7f346c06b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3471e39640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6789/0 conn(0x7f346c0a2620 legacy=0x7f346c1d2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673959938 0 0) 0x7f346c09ff50 con 0x7f346c071420 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): my global_id is 4710 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3450003520 con 0x7f346c071420 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715458453 0 0) 0x7f346c0a0480 con 0x7f346c0a2620 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): my global_id is 4510 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f346c09ff50 con 0x7f346c0a2620 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1481194396 0 0) 0x7f3450003520 con 0x7f346c071420 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f346c0a0480 con 0x7f346c071420 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f345c001320 con 0x7f346c071420 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): dump: 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3774032712 0 0) 0x7f346c09ff50 con 0x7f346c0a2620 2024-09-16T10:30:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3450003520 con 0x7f346c0a2620 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f34640048c0 con 0x7f346c0a2620 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient(hunting): dump: 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 659859801 0 0) 0x7f346c0a0480 con 0x7f346c071420 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.94:6792/0 conn(0x7f346c06ec00 legacy=0x7f346c06b000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6789/0 conn(0x7f346c0a2620 legacy=0x7f346c1d2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 monclient: found mon.g 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:30:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c0a1f60 con 0x7f346c071420 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient: _finish_auth 0 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f347389c640 5 monclient: authenticate success, global_id 4710 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f347389c640 10 monclient: _renew_subs 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f347389c640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346c06cb60 con 0x7f346c071420 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f345c004dd0 con 0x7f346c071420 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f345c005300 con 0x7f346c071420 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:30:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f3461ffb640 10 monclient: dump: 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.749+0000 7f347389c640 10 monclient: _renew_subs 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.749+0000 7f347389c640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.748+0000 7f34637fe640 4 set_mon_vals no callback set 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.749+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f346c127200 con 0x7f346c071420 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.749+0000 7f347389c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.749+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f345c0054e0 con 0x7f346c071420 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.749+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f346c0008d0 con 0x7f346c11d260 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.749+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(21..21 src has 1..21) v4 ==== 3898+0+0 (unknown 3376581462 0 0) 0x7f345c0461a0 con 0x7f346c071420 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.750+0000 7f3471e39640 10 monclient: get_auth_request con 0x7f34500396e0 auth_method 0 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.751+0000 7f3471638640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6792/0 conn(0x7f346c11d260 legacy=0x7f346c11d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41106551 0 0) 0x7f346c0008d0 con 0x7f346c11d260 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 10 monclient(hunting): my global_id is 4552 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f346c0a0480 con 0x7f346c11d260 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 125953631 0 0) 0x7f346c0a0480 con 0x7f346c11d260 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f346c0008d0 con 0x7f346c11d260 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3468003610 con 0x7f346c11d260 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 10 monclient: dump: 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.770 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2552699721 0 0) 0x7f346c0008d0 con 0x7f346c11d260 2024-09-16T10:30:03.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f346c04e730 con 0x7f346c11d260 2024-09-16T10:30:03.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f34680057c0 con 0x7f346c11d260 2024-09-16T10:30:03.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:30:03.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:30:03.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6792/0 conn(0x7f346c11d260 legacy=0x7f346c11d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.752+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6792/0 conn(0x7f346c11d260 legacy=0x7f346c11d640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.758+0000 7f347389c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.758+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f346c11e130 con 0x7f346c11dd50 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3471638640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6792/0 conn(0x7f346c11dd50 legacy=0x7f346c1d8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208693948 0 0) 0x7f346c11e130 con 0x7f346c11dd50 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 10 monclient(hunting): my global_id is 4561 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f346c0008d0 con 0x7f346c11dd50 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3008790074 0 0) 0x7f346c0008d0 con 0x7f346c11dd50 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f346c11e130 con 0x7f346c11dd50 2024-09-16T10:30:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f3468004070 con 0x7f346c11dd50 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 10 monclient: dump: 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:30:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 857692267 0 0) 0x7f346c11e130 con 0x7f346c11dd50 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.761+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 --> v1:172.21.15.152:6792/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f346c125d50 con 0x7f346c11dd50 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 <== mon.7 v1:172.21.15.152:6792/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7441 (unknown 0 0 3176826225) 0x7f3468005930 con 0x7f346c11dd50 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f3461ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f3461ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6792/0 conn(0x7f346c11dd50 legacy=0x7f346c1d8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f3461ffb640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6792/0 conn(0x7f346c11dd50 legacy=0x7f346c1d8320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:30:03.776 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:30:03.776 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:30:03.776 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:30:03.776 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:30:03.776 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 14, 2024-09-16T10:30:03.776 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:30:03.776 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 1, 2024-09-16T10:30:03.777 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.778 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.779 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:30:03.780 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:30:03.781 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.782 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:30:03.783 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.785 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:30:03.786 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.787 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.789 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:30:03.790 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.791 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.792 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:30:03.794 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:30:03.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:30:03.796 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:30:03.797 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:30:03.797 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 3 2024-09-16T10:30:03.797 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:30:03.797 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:30:03.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:30:03.797 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:30:03.797 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:30:03.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6800/2686360781 conn(0x7f34500396e0 legacy=0x7f345003bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f347389c640 10 monclient: shutdown 2024-09-16T10:30:03.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f347389c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:30:03.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 >> v1:172.21.15.152:6790/0 conn(0x7f346c071420 legacy=0x7f346c06b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:30:03.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f3471e39640 1 -- 172.21.15.94:0/3135706554 reap_dead start 2024-09-16T10:30:03.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f3471e39640 1 -- 172.21.15.94:0/3135706554 reap_dead start 2024-09-16T10:30:03.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 shutdown_connections 2024-09-16T10:30:03.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 >> 172.21.15.94:0/3135706554 conn(0x7f346c0a7630 msgr2=0x7f346c0a61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:30:03.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 shutdown_connections 2024-09-16T10:30:03.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:30:03.762+0000 7f347389c640 1 -- 172.21.15.94:0/3135706554 wait complete. 2024-09-16T10:30:03.799 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-16T10:30:03.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T10:30:04.319 INFO:teuthology.orchestra.run.smithi152.stdout:******************************* 2024-09-16T10:30:04.319 INFO:teuthology.orchestra.run.smithi152.stdout:process_4_[34866]: starting. 2024-09-16T10:30:04.319 INFO:teuthology.orchestra.run.smithi152.stdout:process_4_[34866]: deleting pool ceph_test_rados_delete_pools_parallel.smithi152-34830 2024-09-16T10:30:04.319 INFO:teuthology.orchestra.run.smithi152.stdout:process_4_[34866]: shutting down. 2024-09-16T10:30:04.322 INFO:teuthology.orchestra.run.smithi152.stdout:******************************* 2024-09-16T10:30:04.322 INFO:teuthology.orchestra.run.smithi152.stdout:******************************* 2024-09-16T10:30:04.322 INFO:teuthology.orchestra.run.smithi152.stdout:******* SUCCESS ********** 2024-09-16T10:30:04.325 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T10:30:04.335 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T10:31:33.801 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-09-16T10:31:33.801 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-09-16T10:31:33.801 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-16T10:31:33.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-16T10:31:33.806 INFO:tasks.ceph.mon.d:Started 2024-09-16T10:31:33.806 INFO:tasks.mon_thrash:Sending STOP to mon f 2024-09-16T10:31:33.806 INFO:tasks.ceph.mon.f:Sent signal 19 2024-09-16T10:31:33.807 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T10:31:48.809 INFO:tasks.mon_thrash:Sending CONT to mon f 2024-09-16T10:31:48.810 INFO:tasks.ceph.mon.f:Sent signal 18 2024-09-16T10:31:48.810 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-16T10:31:48.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:31:48.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.920+0000 7f8a632b7640 10 monclient: get_monmap_and_config 2024-09-16T10:31:48.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.920+0000 7f8a632b7640 10 monclient: build_initial_monmap 2024-09-16T10:31:48.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.920+0000 7f8a632b7640 10 monclient: monmap: 2024-09-16T10:31:48.922 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:48.921374+0000 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:48.921374+0000 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:48.923 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:48.924 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:48.924 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:48.924 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:48.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 1 Processor -- start 2024-09-16T10:31:48.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 1 -- start start 2024-09-16T10:31:48.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 10 monclient: init 2024-09-16T10:31:48.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:48.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 10 monclient: _add_conns ranks=[4,7,6,5,0,8,1,3,2] 2024-09-16T10:31:48.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 10 monclient(hunting): picked mon.noname-c con 0x7f8a5c1368e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:48.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 10 monclient(hunting): picked mon.noname-i con 0x7f8a5c136cc0 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:48.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 10 monclient(hunting): picked mon.noname-d con 0x7f8a5c1370a0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:48.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c061ae0 con 0x7f8a5c1368e0 2024-09-16T10:31:48.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c0a03e0 con 0x7f8a5c1370a0 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c06c8d0 con 0x7f8a5c136cc0 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a632b7640 10 monclient(hunting): authenticate will time out at 876.261898s 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.921+0000 7f8a61053640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f8a5c1368e0 0x7f8a5c0a1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33954/0 (socket says 172.21.15.94:33954) 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.922+0000 7f8a61053640 1 -- 172.21.15.94:0/1447582272 learned_addr learned my addr 172.21.15.94:0/1447582272 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a61854640 1 -- 172.21.15.94:0/1447582272 >> v1:172.21.15.94:6792/0 conn(0x7f8a5c1370a0 legacy=0x7f8a5c13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a61053640 1 -- 172.21.15.94:0/1447582272 >> v1:172.21.15.94:6791/0 conn(0x7f8a5c1368e0 legacy=0x7f8a5c0a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946226567 0 0) 0x7f8a5c061ae0 con 0x7f8a5c1368e0 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:48.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient(hunting): my global_id is 4567 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a48003520 con 0x7f8a5c1368e0 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a60852640 1 -- 172.21.15.94:0/1447582272 >> v1:172.21.15.152:6792/0 conn(0x7f8a5c136cc0 legacy=0x7f8a5c13b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852115584 0 0) 0x7f8a5c0a03e0 con 0x7f8a5c1370a0 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient(hunting): my global_id is 94110 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c061ae0 con 0x7f8a5c1370a0 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3362490158 0 0) 0x7f8a48003520 con 0x7f8a5c1368e0 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 >> v1:172.21.15.152:6792/0 conn(0x7f8a5c136cc0 legacy=0x7f8a5c13b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:48.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 >> v1:172.21.15.94:6792/0 conn(0x7f8a5c1370a0 legacy=0x7f8a5c13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: _finish_hunting 0 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 monclient: found mon.noname-c 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c0a0680 con 0x7f8a5c1368e0 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: _finish_auth 0 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: _check_auth_tickets 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8a58001990 con 0x7f8a5c1368e0 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:48.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: dump: 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:48.933 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a632b7640 5 monclient: authenticate success, global_id 4567 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a632b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a58001ca0 con 0x7f8a5c1368e0 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 1 -- 172.21.15.94:0/1447582272 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8a58004be0 con 0x7f8a5c1368e0 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a632b7640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a632b7640 4 set_mon_vals no callback set 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:48.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.924+0000 7f8a53fff640 10 monclient: dump: 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.925+0000 7f8a632b7640 10 monclient: shutdown 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.925+0000 7f8a632b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.925+0000 7f8a632b7640 1 -- 172.21.15.94:0/1447582272 >> v1:172.21.15.94:6791/0 conn(0x7f8a5c1368e0 legacy=0x7f8a5c0a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.925+0000 7f8a632b7640 1 -- 172.21.15.94:0/1447582272 shutdown_connections 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.925+0000 7f8a632b7640 1 -- 172.21.15.94:0/1447582272 >> 172.21.15.94:0/1447582272 conn(0x7f8a5c0a58d0 msgr2=0x7f8a5c0a5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.925+0000 7f8a632b7640 1 -- 172.21.15.94:0/1447582272 shutdown_connections 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.925+0000 7f8a632b7640 1 -- 172.21.15.94:0/1447582272 wait complete. 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.925+0000 7f8a632b7640 10 monclient: build_initial_monmap 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.925+0000 7f8a632b7640 10 monclient: monmap: 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:48.936 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:48.937 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 1 Processor -- start 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 1 -- start start 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 10 monclient: init 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 10 monclient: _add_conns ranks=[3,8,7,0,2,6,1,4,5] 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 10 monclient(hunting): picked mon.noname-g con 0x7f8a5c1368e0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 10 monclient(hunting): picked mon.noname-e con 0x7f8a5c136cc0 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 10 monclient(hunting): picked mon.noname-i con 0x7f8a5c1370a0 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c123720 con 0x7f8a5c1368e0 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c0a28f0 con 0x7f8a5c1370a0 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c122000 con 0x7f8a5c136cc0 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a632b7640 10 monclient(hunting): authenticate will time out at 876.266903s 2024-09-16T10:31:48.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a60852640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f8a5c136cc0 0x7f8a5c11ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:43948/0 (socket says 172.21.15.94:43948) 2024-09-16T10:31:48.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.926+0000 7f8a60852640 1 -- 172.21.15.94:0/3548929229 learned_addr learned my addr 172.21.15.94:0/3548929229 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a60852640 1 -- 172.21.15.94:0/3548929229 >> v1:172.21.15.94:6793/0 conn(0x7f8a5c136cc0 legacy=0x7f8a5c11ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a61854640 1 -- 172.21.15.94:0/3548929229 >> v1:172.21.15.152:6792/0 conn(0x7f8a5c1370a0 legacy=0x7f8a5c1d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a61053640 1 -- 172.21.15.94:0/3548929229 >> v1:172.21.15.152:6790/0 conn(0x7f8a5c1368e0 legacy=0x7f8a5c11f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702729005 0 0) 0x7f8a5c0a28f0 con 0x7f8a5c1370a0 2024-09-16T10:31:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): my global_id is 4579 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a40003520 con 0x7f8a5c1370a0 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719899797 0 0) 0x7f8a5c123720 con 0x7f8a5c1368e0 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): my global_id is 4719 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c0a28f0 con 0x7f8a5c1368e0 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3807550951 0 0) 0x7f8a40003520 con 0x7f8a5c1370a0 2024-09-16T10:31:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a5c123720 con 0x7f8a5c1370a0 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8a54003cd0 con 0x7f8a5c1370a0 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): dump: 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:48.944 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2135110467 0 0) 0x7f8a5c0a28f0 con 0x7f8a5c1368e0 2024-09-16T10:31:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a40003520 con 0x7f8a5c1368e0 2024-09-16T10:31:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8a58004b20 con 0x7f8a5c1368e0 2024-09-16T10:31:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:31:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 10 monclient(hunting): dump: 2024-09-16T10:31:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.929+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2012317424 0 0) 0x7f8a5c123720 con 0x7f8a5c1370a0 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 >> v1:172.21.15.94:6793/0 conn(0x7f8a5c136cc0 legacy=0x7f8a5c11ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 >> v1:172.21.15.152:6790/0 conn(0x7f8a5c1368e0 legacy=0x7f8a5c11f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:48.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:31:48.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 1 monclient: found mon.i 2024-09-16T10:31:48.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:48.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:31:48.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 --> v1:172.21.15.152:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c0a24a0 con 0x7f8a5c1370a0 2024-09-16T10:31:48.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 10 monclient: _finish_auth 0 2024-09-16T10:31:48.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:31:48.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a632b7640 5 monclient: authenticate success, global_id 4579 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a632b7640 10 monclient: _renew_subs 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a632b7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 --> v1:172.21.15.152:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c123440 con 0x7f8a5c1370a0 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a54003f30 con 0x7f8a5c1370a0 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8a540043e0 con 0x7f8a5c1370a0 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a51ffb640 10 monclient: dump: 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a537fe640 4 set_mon_vals no callback set 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a632b7640 10 monclient: _renew_subs 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a632b7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.930+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 --> v1:172.21.15.152:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c120630 con 0x7f8a5c1370a0 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.931+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f8a54004640 con 0x7f8a5c1370a0 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.931+0000 7f8a632b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.931+0000 7f8a632b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.931+0000 7f8a632b7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:31:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.931+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c06c850 con 0x7f8a5c1370a0 2024-09-16T10:31:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.931+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f8a54046210 con 0x7f8a5c1370a0 2024-09-16T10:31:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.931+0000 7f8a61053640 10 monclient: get_auth_request con 0x7f8a400397c0 auth_method 0 2024-09-16T10:31:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.936+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f8a54015720 con 0x7f8a5c1370a0 2024-09-16T10:31:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.936+0000 7f8a51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:48.936+0000 7f8a51ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:49.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.163+0000 7f8a632b7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:31:49.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.163+0000 7f8a632b7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:31:49.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.163+0000 7f8a632b7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.152:6792/0 2024-09-16T10:31:49.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.163+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 --> v1:172.21.15.152:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8a5c0a0cd0 con 0x7f8a5c1370a0 2024-09-16T10:31:49.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.163+0000 7f8a51ffb640 1 -- 172.21.15.94:0/3548929229 <== mon.7 v1:172.21.15.152:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+2712 (unknown 254746657 0 3369823156) 0x7f8a54015310 con 0x7f8a5c1370a0 2024-09-16T10:31:49.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.164+0000 7f8a51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:31:49.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.164+0000 7f8a51ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:49.165 INFO:teuthology.orchestra.run.smithi094.stdout:{"election_epoch":18,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","modified":"2024-09-16T10:28:46.257482Z","created":"2024-09-16T10:28:46.257482Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6790","nonce":0}]},"addr":"172.21.15.94:6790/0","public_addr":"172.21.15.94:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6790","nonce":0}]},"addr":"172.21.15.152:6790/0","public_addr":"172.21.15.152:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6791","nonce":0}]},"addr":"172.21.15.94:6791/0","public_addr":"172.21.15.94:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6791","nonce":0}]},"addr":"172.21.15.152:6791/0","public_addr":"172.21.15.152:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6792","nonce":0}]},"addr":"172.21.15.94:6792/0","public_addr":"172.21.15.94:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6792","nonce":0}]},"addr":"172.21.15.152:6792/0","public_addr":"172.21.15.152:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6793","nonce":0}]},"addr":"172.21.15.94:6793/0","public_addr":"172.21.15.94:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:31:49.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.165+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8a400397c0 legacy=0x7f8a4003bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.166+0000 7f8a632b7640 10 monclient: shutdown 2024-09-16T10:31:49.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.166+0000 7f8a632b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.166+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 >> v1:172.21.15.152:6792/0 conn(0x7f8a5c1370a0 legacy=0x7f8a5c1d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.166+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 shutdown_connections 2024-09-16T10:31:49.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.166+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 >> 172.21.15.94:0/3548929229 conn(0x7f8a5c0a58d0 msgr2=0x7f8a5c1be2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.166+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 shutdown_connections 2024-09-16T10:31:49.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.166+0000 7f8a632b7640 1 -- 172.21.15.94:0/3548929229 wait complete. 2024-09-16T10:31:49.179 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-16T10:31:49.179 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-16T10:31:49.179 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:31:49.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.274+0000 7f92da101640 10 monclient: get_monmap_and_config 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.274+0000 7f92da101640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.274+0000 7f92da101640 10 monclient: monmap: 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:49.275393+0000 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:49.275393+0000 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.276 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.277 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.277 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.277 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.277 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.277 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.277 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.277 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 1 Processor -- start 2024-09-16T10:31:49.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 1 -- start start 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 10 monclient: init 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 10 monclient: _add_conns ranks=[2,3,8,0,5,6,4,7,1] 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 10 monclient(hunting): picked mon.noname-b con 0x7f92d40a3860 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 10 monclient(hunting): picked mon.noname-g con 0x7f92d406c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 10 monclient(hunting): picked mon.noname-e con 0x7f92d40a4350 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d4061ae0 con 0x7f92d40a3860 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d40a0600 con 0x7f92d406c1d0 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d40a0b30 con 0x7f92d40a4350 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92da101640 10 monclient(hunting): authenticate will time out at 876.615867s 2024-09-16T10:31:49.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92d37fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f92d40a3860 0x7f92d40a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41040/0 (socket says 172.21.15.94:41040) 2024-09-16T10:31:49.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92d3fff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f92d40a4350 0x7f92d413ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:43964/0 (socket says 172.21.15.94:43964) 2024-09-16T10:31:49.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.275+0000 7f92d37fe640 1 -- 172.21.15.94:0/933546858 learned_addr learned my addr 172.21.15.94:0/933546858 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d37fe640 1 -- 172.21.15.94:0/933546858 >> v1:172.21.15.94:6790/0 conn(0x7f92d40a3860 legacy=0x7f92d40a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d3fff640 1 -- 172.21.15.94:0/933546858 >> v1:172.21.15.94:6793/0 conn(0x7f92d40a4350 legacy=0x7f92d413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728785177 0 0) 0x7f92d4061ae0 con 0x7f92d40a3860 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient(hunting): my global_id is 4529 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92bc003520 con 0x7f92d40a3860 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474542094 0 0) 0x7f92d40a0b30 con 0x7f92d40a4350 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient(hunting): my global_id is 4535 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d4061ae0 con 0x7f92d40a4350 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d2ffd640 1 -- 172.21.15.94:0/933546858 >> v1:172.21.15.152:6790/0 conn(0x7f92d406c1d0 legacy=0x7f92d413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1509246138 0 0) 0x7f92bc003520 con 0x7f92d40a3860 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 >> v1:172.21.15.94:6793/0 conn(0x7f92d40a4350 legacy=0x7f92d413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 >> v1:172.21.15.152:6790/0 conn(0x7f92d406c1d0 legacy=0x7f92d413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 monclient: found mon.noname-b 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:31:49.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d40a0e10 con 0x7f92d40a3860 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f92c4001a50 con 0x7f92d40a3860 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92da101640 5 monclient: authenticate success, global_id 4529 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92da101640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient: dump: 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92da101640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d37fe640 1 --1- 172.21.15.94:0/933546858 >> v1:172.21.15.94:6790/0 conn(0x7f92d40a3860 0x7f92d40a3c40 :-1 s=OPENED pgs=150 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7f92c4004be0 mon_map magic: 0 v1 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92c4001d60 con 0x7f92d40a3860 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92d27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92da101640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.278+0000 7f92da101640 4 set_mon_vals no callback set 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92da101640 10 monclient: shutdown 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92da101640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92da101640 1 -- 172.21.15.94:0/933546858 >> v1:172.21.15.94:6790/0 conn(0x7f92d40a3860 legacy=0x7f92d40a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92d27fc640 1 -- 172.21.15.94:0/933546858 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f92c4004be0 con 0x7f92d40a3860 2024-09-16T10:31:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92d27fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92da101640 1 -- 172.21.15.94:0/933546858 shutdown_connections 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92da101640 1 -- 172.21.15.94:0/933546858 >> 172.21.15.94:0/933546858 conn(0x7f92d40a5960 msgr2=0x7f92d40a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92da101640 1 -- 172.21.15.94:0/933546858 shutdown_connections 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92da101640 1 -- 172.21.15.94:0/933546858 wait complete. 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92da101640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.279+0000 7f92da101640 10 monclient: monmap: 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.289 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 1 Processor -- start 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 1 -- start start 2024-09-16T10:31:49.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 10 monclient: init 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 10 monclient: _add_conns ranks=[3,7,8,0,5,4,6,1,2] 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 10 monclient(hunting): picked mon.noname-g con 0x7f92d406c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 10 monclient(hunting): picked mon.noname-i con 0x7f92d40a3860 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 10 monclient(hunting): picked mon.noname-e con 0x7f92d40a4350 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d41a77b0 con 0x7f92d406c1d0 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d409f7e0 con 0x7f92d40a3860 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d41a9180 con 0x7f92d40a4350 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92da101640 10 monclient(hunting): authenticate will time out at 876.620779s 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92d3fff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f92d40a4350 0x7f92d41d3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:43978/0 (socket says 172.21.15.94:43978) 2024-09-16T10:31:49.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.280+0000 7f92d3fff640 1 -- 172.21.15.94:0/1289309979 learned_addr learned my addr 172.21.15.94:0/1289309979 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92d3fff640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.94:6793/0 conn(0x7f92d40a4350 legacy=0x7f92d41d3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92d2ffd640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.152:6792/0 conn(0x7f92d40a3860 legacy=0x7f92d41cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92d37fe640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.152:6790/0 conn(0x7f92d406c1d0 legacy=0x7f92d41bb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899866114 0 0) 0x7f92d41a9180 con 0x7f92d40a4350 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): my global_id is 4544 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c0003550 con 0x7f92d40a4350 2024-09-16T10:31:49.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3868530653 0 0) 0x7f92c0003550 con 0x7f92d40a4350 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d41a9180 con 0x7f92d40a4350 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f92cc0020d0 con 0x7f92d40a4350 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): dump: 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.297 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809171316 0 0) 0x7f92d41a77b0 con 0x7f92d406c1d0 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): my global_id is 4737 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c0003550 con 0x7f92d406c1d0 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600978153 0 0) 0x7f92d409f7e0 con 0x7f92d40a3860 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient(hunting): my global_id is 4588 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d41a77b0 con 0x7f92d40a3860 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3207621502 0 0) 0x7f92d41a9180 con 0x7f92d40a4350 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.152:6792/0 conn(0x7f92d40a3860 legacy=0x7f92d41cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.152:6790/0 conn(0x7f92d406c1d0 legacy=0x7f92d41bb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 monclient: found mon.e 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d41a6db0 con 0x7f92d40a4350 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92da101640 5 monclient: authenticate success, global_id 4544 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92da101640 10 monclient: _renew_subs 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92da101640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6793/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d409f290 con 0x7f92d40a4350 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92da101640 10 monclient: _renew_subs 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92da101640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d41a8ea0 con 0x7f92d40a4350 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.8 v1:172.21.15.94:6793/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92cc005150 con 0x7f92d40a4350 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.8 v1:172.21.15.94:6793/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f92cc005600 con 0x7f92d40a4350 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92bbfff640 10 monclient: dump: 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.301 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.283+0000 7f92d1ffb640 4 set_mon_vals no callback set 2024-09-16T10:31:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.284+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.8 v1:172.21.15.94:6793/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f92cc010660 con 0x7f92d40a4350 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.284+0000 7f92da101640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.284+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f92d41bc2b0 con 0x7f92d41bbed0 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.284+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.8 v1:172.21.15.94:6793/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f92cc046310 con 0x7f92d40a4350 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.285+0000 7f92d2ffd640 10 monclient: get_auth_request con 0x7f92c00392b0 auth_method 0 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.286+0000 7f92d37fe640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.94:6789/0 conn(0x7f92d41bbed0 legacy=0x7f92d409b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.286+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139046475 0 0) 0x7f92d41bc2b0 con 0x7f92d41bbed0 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.286+0000 7f92bbfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.286+0000 7f92bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.286+0000 7f92bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.286+0000 7f92bbfff640 10 monclient(hunting): my global_id is 4527 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d41a9180 con 0x7f92d41bbed0 2024-09-16T10:31:49.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2817738972 0 0) 0x7f92d41a9180 con 0x7f92d41bbed0 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d41bc2b0 con 0x7f92d41bbed0 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f92c4004b20 con 0x7f92d41bbed0 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 10 monclient: dump: 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.305 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.305 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.305 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.305 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.305 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.305 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.305 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.305 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3770890388 0 0) 0x7f92d41bc2b0 con 0x7f92d41bbed0 2024-09-16T10:31:49.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92d40a4920 con 0x7f92d41bbed0 2024-09-16T10:31:49.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f92c4001800 con 0x7f92d41bbed0 2024-09-16T10:31:49.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:49.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.94:6789/0 conn(0x7f92d41bbed0 legacy=0x7f92d409b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.287+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.94:6789/0 conn(0x7f92d41bbed0 legacy=0x7f92d409b120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.293+0000 7f92da101640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:31:49.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.293+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f92d41cb2f0 con 0x7f92d41caf10 2024-09-16T10:31:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92d37fe640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.94:6789/0 conn(0x7f92d41caf10 legacy=0x7f92d409b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288176930 0 0) 0x7f92d41cb2f0 con 0x7f92d41caf10 2024-09-16T10:31:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 10 monclient(hunting): my global_id is 4536 2024-09-16T10:31:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d41bc2b0 con 0x7f92d41caf10 2024-09-16T10:31:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2138888495 0 0) 0x7f92d41bc2b0 con 0x7f92d41caf10 2024-09-16T10:31:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d41cb2f0 con 0x7f92d41caf10 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f92c4005340 con 0x7f92d41caf10 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 10 monclient: dump: 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2006310231 0 0) 0x7f92d41cb2f0 con 0x7f92d41caf10 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 --> v1:172.21.15.94:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f92d404e730 con 0x7f92d41caf10 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 <== mon.0 v1:172.21.15.94:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7454 (unknown 0 0 2808506638) 0x7f92c40075e0 con 0x7f92d41caf10 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.94:6789/0 conn(0x7f92d41caf10 legacy=0x7f92d409b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.296+0000 7f92bbfff640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.94:6789/0 conn(0x7f92d41caf10 legacy=0x7f92d409b830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "leader", 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 18, 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:31:49.311 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 0, 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:31:49.312 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:31:49.313 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:31:49.315 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.316 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.317 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.319 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.321 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.322 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.323 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.324 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.325 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.326 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.328 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.330 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.331 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:31:49.332 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:49.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.152:6800/2686360781 conn(0x7f92c00392b0 legacy=0x7f92c003b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92da101640 10 monclient: shutdown 2024-09-16T10:31:49.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92da101640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 >> v1:172.21.15.94:6793/0 conn(0x7f92d40a4350 legacy=0x7f92d41d3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92d3fff640 1 -- 172.21.15.94:0/1289309979 reap_dead start 2024-09-16T10:31:49.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92d3fff640 1 -- 172.21.15.94:0/1289309979 reap_dead start 2024-09-16T10:31:49.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 shutdown_connections 2024-09-16T10:31:49.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 >> 172.21.15.94:0/1289309979 conn(0x7f92d40a5960 msgr2=0x7f92d40a7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 shutdown_connections 2024-09-16T10:31:49.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.297+0000 7f92da101640 1 -- 172.21.15.94:0/1289309979 wait complete. 2024-09-16T10:31:49.334 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:31:49.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-16T10:31:49.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:31:49.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.429+0000 7f4578d9d640 10 monclient: get_monmap_and_config 2024-09-16T10:31:49.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.429+0000 7f4578d9d640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.429+0000 7f4578d9d640 10 monclient: monmap: 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:49.430725+0000 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:49.430725+0000 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.433 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.433 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.433 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.433 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.430+0000 7f4578d9d640 1 Processor -- start 2024-09-16T10:31:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.430+0000 7f4578d9d640 1 -- start start 2024-09-16T10:31:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.430+0000 7f4578d9d640 10 monclient: init 2024-09-16T10:31:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 10 monclient: _add_conns ranks=[5,3,4,6,1,0,7,2,8] 2024-09-16T10:31:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 10 monclient(hunting): picked mon.noname-h con 0x7f4574132e30 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 10 monclient(hunting): picked mon.noname-g con 0x7f45740a4a90 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:49.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f457413c960 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:49.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457404f100 con 0x7f45740a4a90 2024-09-16T10:31:49.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457409aa60 con 0x7f457413c960 2024-09-16T10:31:49.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457409ad40 con 0x7f4574132e30 2024-09-16T10:31:49.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4578d9d640 10 monclient(hunting): authenticate will time out at 876.771227s 2024-09-16T10:31:49.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4573fff640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f457413c960 0x7f457413ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33958/0 (socket says 172.21.15.94:33958) 2024-09-16T10:31:49.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.431+0000 7f4573fff640 1 -- 172.21.15.94:0/145510536 learned_addr learned my addr 172.21.15.94:0/145510536 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f4573fff640 1 -- 172.21.15.94:0/145510536 >> v1:172.21.15.94:6791/0 conn(0x7f457413c960 legacy=0x7f457413ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 1 -- 172.21.15.94:0/145510536 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305607498 0 0) 0x7f457409aa60 con 0x7f457413c960 2024-09-16T10:31:49.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient(hunting): my global_id is 4576 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 1 -- 172.21.15.94:0/145510536 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c003520 con 0x7f457413c960 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45737fe640 1 -- 172.21.15.94:0/145510536 >> v1:172.21.15.152:6791/0 conn(0x7f4574132e30 legacy=0x7f4574133210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f4572ffd640 1 -- 172.21.15.94:0/145510536 >> v1:172.21.15.152:6790/0 conn(0x7f45740a4a90 legacy=0x7f457413b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 1 -- 172.21.15.94:0/145510536 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1644867588 0 0) 0x7f455c003520 con 0x7f457413c960 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 1 -- 172.21.15.94:0/145510536 >> v1:172.21.15.152:6791/0 conn(0x7f4574132e30 legacy=0x7f4574133210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 1 -- 172.21.15.94:0/145510536 >> v1:172.21.15.152:6790/0 conn(0x7f45740a4a90 legacy=0x7f457413b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 1 monclient: found mon.noname-c 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:31:49.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 1 -- 172.21.15.94:0/145510536 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457409b020 con 0x7f457413c960 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f4578d9d640 5 monclient: authenticate success, global_id 4576 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 1 -- 172.21.15.94:0/145510536 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f456c001a60 con 0x7f457413c960 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f4578d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f45727fc640 10 monclient: dump: 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.433+0000 7f4578d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f45727fc640 1 -- 172.21.15.94:0/145510536 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f456c001d70 con 0x7f457413c960 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f45727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f45727fc640 1 -- 172.21.15.94:0/145510536 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f456c004be0 con 0x7f457413c960 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f45727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f45727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f45727fc640 10 monclient: dump: 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.442 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f4578d9d640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:49.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f4578d9d640 4 set_mon_vals no callback set 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f4578d9d640 10 monclient: shutdown 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f4578d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f4578d9d640 1 -- 172.21.15.94:0/145510536 >> v1:172.21.15.94:6791/0 conn(0x7f457413c960 legacy=0x7f457413ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f4578d9d640 1 -- 172.21.15.94:0/145510536 shutdown_connections 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f4578d9d640 1 -- 172.21.15.94:0/145510536 >> 172.21.15.94:0/145510536 conn(0x7f45740a93f0 msgr2=0x7f45740a5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f4578d9d640 1 -- 172.21.15.94:0/145510536 shutdown_connections 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.434+0000 7f4578d9d640 1 -- 172.21.15.94:0/145510536 wait complete. 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.435+0000 7f4578d9d640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.435+0000 7f4578d9d640 10 monclient: monmap: 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:49.444 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.435+0000 7f4578d9d640 1 Processor -- start 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.435+0000 7f4578d9d640 1 -- start start 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.435+0000 7f4578d9d640 10 monclient: init 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 10 monclient: _add_conns ranks=[2,1,3,4,5,6,8,0,7] 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f45740a4a90 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 10 monclient(hunting): picked mon.noname-f con 0x7f4574132e30 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 10 monclient(hunting): picked mon.noname-g con 0x7f457413c960 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4574131090 con 0x7f4574132e30 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45741cb900 con 0x7f45740a4a90 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457413c2b0 con 0x7f457413c960 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f4578d9d640 10 monclient(hunting): authenticate will time out at 876.776116s 2024-09-16T10:31:49.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f45737fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f45740a4a90 0x7f45741caae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41044/0 (socket says 172.21.15.94:41044) 2024-09-16T10:31:49.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.436+0000 7f45737fe640 1 -- 172.21.15.94:0/1367096861 learned_addr learned my addr 172.21.15.94:0/1367096861 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f45737fe640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.94:6790/0 conn(0x7f45740a4a90 legacy=0x7f45741caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f4573fff640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.152:6790/0 conn(0x7f457413c960 legacy=0x7f45741d2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953318297 0 0) 0x7f45741cb900 con 0x7f45740a4a90 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): my global_id is 4538 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560003520 con 0x7f45740a4a90 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f4572ffd640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.152:6789/0 conn(0x7f4574132e30 legacy=0x7f45741cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403201782 0 0) 0x7f457413c2b0 con 0x7f457413c960 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): my global_id is 4755 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45741cb900 con 0x7f457413c960 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 684500180 0 0) 0x7f4560003520 con 0x7f45740a4a90 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457413c2b0 con 0x7f45740a4a90 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f45640020b0 con 0x7f45740a4a90 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): dump: 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.452 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.453 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966103791 0 0) 0x7f4574131090 con 0x7f4574132e30 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient(hunting): my global_id is 4519 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560003520 con 0x7f4574132e30 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3064322088 0 0) 0x7f457413c2b0 con 0x7f45740a4a90 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.152:6790/0 conn(0x7f457413c960 legacy=0x7f45741d2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.152:6789/0 conn(0x7f4574132e30 legacy=0x7f45741cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 monclient: found mon.b 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457409e8a0 con 0x7f45740a4a90 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f455bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f4578d9d640 5 monclient: authenticate success, global_id 4538 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f4578d9d640 10 monclient: _renew_subs 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f4578d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457413b670 con 0x7f45740a4a90 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f4578d9d640 10 monclient: _renew_subs 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f4578d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.438+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f457413ad20 con 0x7f45740a4a90 2024-09-16T10:31:49.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4564003ef0 con 0x7f45740a4a90 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f455bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f45640043a0 con 0x7f45740a4a90 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f455bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f4571ffb640 4 set_mon_vals no callback set 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f455bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f455bfff640 10 monclient: dump: 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.456 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f4578d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4554006580 con 0x7f4554002820 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f4564004580 con 0x7f45740a4a90 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.439+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f45640462c0 con 0x7f45740a4a90 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.440+0000 7f4573fff640 10 monclient: get_auth_request con 0x7f4560039240 auth_method 0 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.441+0000 7f4572ffd640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.94:6790/0 conn(0x7f4554002820 legacy=0x7f4554004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041295496 0 0) 0x7f4554006580 con 0x7f4554002820 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 10 monclient(hunting): my global_id is 4547 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457413c2b0 con 0x7f4554002820 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 522244578 0 0) 0x7f457413c2b0 con 0x7f4554002820 2024-09-16T10:31:49.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4554006580 con 0x7f4554002820 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4568001c70 con 0x7f4554002820 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 10 monclient: dump: 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.459 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 937370148 0 0) 0x7f4554006580 con 0x7f4554002820 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f455400a0d0 con 0x7f4554002820 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f45680068b0 con 0x7f4554002820 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:49.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.94:6790/0 conn(0x7f4554002820 legacy=0x7f4554004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.442+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.94:6790/0 conn(0x7f4554002820 legacy=0x7f4554004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.448+0000 7f4578d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:31:49.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.449+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4554009d70 con 0x7f4554009090 2024-09-16T10:31:49.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f4572ffd640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.94:6790/0 conn(0x7f4554009090 legacy=0x7f4554009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951520455 0 0) 0x7f4554009d70 con 0x7f4554009090 2024-09-16T10:31:49.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 10 monclient(hunting): my global_id is 4556 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4554006580 con 0x7f4554009090 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2429377975 0 0) 0x7f4554006580 con 0x7f4554009090 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4554009d70 con 0x7f4554009090 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f4568003e00 con 0x7f4554009090 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 10 monclient: dump: 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.463 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3454022453 0 0) 0x7f4554009d70 con 0x7f4554009090 2024-09-16T10:31:49.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 --> v1:172.21.15.94:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f455400c280 con 0x7f4554009090 2024-09-16T10:31:49.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 <== mon.2 v1:172.21.15.94:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7784 (unknown 0 0 779772236) 0x7f4568004320 con 0x7f4554009090 2024-09-16T10:31:49.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:31:49.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:49.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.94:6790/0 conn(0x7f4554009090 legacy=0x7f4554009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.451+0000 7f455bfff640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.94:6790/0 conn(0x7f4554009090 legacy=0x7f4554009520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 18, 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 0, 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:31:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": [ 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:31:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:49.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.152:6800/2686360781 conn(0x7f4560039240 legacy=0x7f456003b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4578d9d640 10 monclient: shutdown 2024-09-16T10:31:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4578d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 >> v1:172.21.15.94:6790/0 conn(0x7f45740a4a90 legacy=0x7f45741caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4573fff640 1 -- 172.21.15.94:0/1367096861 reap_dead start 2024-09-16T10:31:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4573fff640 1 -- 172.21.15.94:0/1367096861 reap_dead start 2024-09-16T10:31:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 shutdown_connections 2024-09-16T10:31:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 >> 172.21.15.94:0/1367096861 conn(0x7f45740a93f0 msgr2=0x7f45740a7300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 shutdown_connections 2024-09-16T10:31:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.452+0000 7f4578d9d640 1 -- 172.21.15.94:0/1367096861 wait complete. 2024-09-16T10:31:49.489 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:31:49.489 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-16T10:31:49.489 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.584+0000 7f871019c640 10 monclient: get_monmap_and_config 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.584+0000 7f871019c640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.584+0000 7f871019c640 10 monclient: monmap: 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:49.585378+0000 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:49.585378+0000 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.587 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 1 Processor -- start 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 1 -- start start 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 10 monclient: init 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 10 monclient: _add_conns ranks=[7,0,5,8,1,3,6,4,2] 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 10 monclient(hunting): picked mon.noname-i con 0x7f87080a3860 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 10 monclient(hunting): picked mon.noname-a con 0x7f870806c1d0 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 10 monclient(hunting): picked mon.noname-h con 0x7f87080a4350 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8708061ae0 con 0x7f870806c1d0 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87080a0600 con 0x7f87080a4350 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87080a0b30 con 0x7f87080a3860 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f871019c640 10 monclient(hunting): authenticate will time out at 876.925844s 2024-09-16T10:31:49.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f870d737640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f870806c1d0 0x7f870813b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53428/0 (socket says 172.21.15.94:53428) 2024-09-16T10:31:49.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.585+0000 7f870d737640 1 -- 172.21.15.94:0/1850691646 learned_addr learned my addr 172.21.15.94:0/1850691646 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870d737640 1 -- 172.21.15.94:0/1850691646 >> v1:172.21.15.94:6789/0 conn(0x7f870806c1d0 legacy=0x7f870813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870df38640 1 -- 172.21.15.94:0/1850691646 >> v1:172.21.15.152:6792/0 conn(0x7f87080a3860 legacy=0x7f87080a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363985060 0 0) 0x7f8708061ae0 con 0x7f870806c1d0 2024-09-16T10:31:49.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): my global_id is 4545 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870e739640 1 -- 172.21.15.94:0/1850691646 >> v1:172.21.15.152:6791/0 conn(0x7f87080a4350 legacy=0x7f870813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f4003520 con 0x7f870806c1d0 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036181829 0 0) 0x7f87080a0600 con 0x7f87080a4350 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): my global_id is 4595 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8708061ae0 con 0x7f87080a4350 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114875221 0 0) 0x7f87080a0b30 con 0x7f87080a3860 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient(hunting): my global_id is 4597 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87080a0600 con 0x7f87080a3860 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447009935 0 0) 0x7f86f4003520 con 0x7f870806c1d0 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 >> v1:172.21.15.152:6792/0 conn(0x7f87080a3860 legacy=0x7f87080a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 >> v1:172.21.15.152:6791/0 conn(0x7f87080a4350 legacy=0x7f870813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 monclient: found mon.noname-a 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87080a0e10 con 0x7f870806c1d0 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f86f8001a50 con 0x7f870806c1d0 2024-09-16T10:31:49.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f871019c640 5 monclient: authenticate success, global_id 4545 2024-09-16T10:31:49.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f871019c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:49.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:31:49.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: dump: 2024-09-16T10:31:49.596 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.596 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.596 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.597 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f871019c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86f8001ce0 con 0x7f870806c1d0 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f871019c640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 1 -- 172.21.15.94:0/1850691646 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f86f8004be0 con 0x7f870806c1d0 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f871019c640 4 set_mon_vals no callback set 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.588+0000 7f870cf36640 10 monclient: dump: 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.598 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.599 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.589+0000 7f871019c640 10 monclient: shutdown 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.589+0000 7f871019c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.589+0000 7f871019c640 1 -- 172.21.15.94:0/1850691646 >> v1:172.21.15.94:6789/0 conn(0x7f870806c1d0 legacy=0x7f870813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.589+0000 7f871019c640 1 -- 172.21.15.94:0/1850691646 shutdown_connections 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.589+0000 7f871019c640 1 -- 172.21.15.94:0/1850691646 >> 172.21.15.94:0/1850691646 conn(0x7f87080a5960 msgr2=0x7f87080a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.589+0000 7f871019c640 1 -- 172.21.15.94:0/1850691646 shutdown_connections 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.589+0000 7f871019c640 1 -- 172.21.15.94:0/1850691646 wait complete. 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.589+0000 7f871019c640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.589+0000 7f871019c640 10 monclient: monmap: 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:49.600 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.601 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 1 Processor -- start 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 1 -- start start 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 10 monclient: init 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 10 monclient: _add_conns ranks=[8,7,1,4,3,0,5,6,2] 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 10 monclient(hunting): picked mon.noname-e con 0x7f870806c1d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 10 monclient(hunting): picked mon.noname-i con 0x7f87080a3860 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 10 monclient(hunting): picked mon.noname-f con 0x7f87080a4350 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87081a9a40 con 0x7f87080a4350 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870809f730 con 0x7f87080a3860 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870809f200 con 0x7f870806c1d0 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f871019c640 10 monclient(hunting): authenticate will time out at 876.930534s 2024-09-16T10:31:49.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f870df38640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f870806c1d0 0x7f87081bd810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:43992/0 (socket says 172.21.15.94:43992) 2024-09-16T10:31:49.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.590+0000 7f870df38640 1 -- 172.21.15.94:0/2416792159 learned_addr learned my addr 172.21.15.94:0/2416792159 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.592+0000 7f870df38640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.94:6793/0 conn(0x7f870806c1d0 legacy=0x7f87081bd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.592+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851834009 0 0) 0x7f870809f200 con 0x7f870806c1d0 2024-09-16T10:31:49.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.592+0000 7f86fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.592+0000 7f86fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.592+0000 7f86fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.592+0000 7f86fe7fc640 10 monclient(hunting): my global_id is 4553 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.592+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ec003520 con 0x7f870806c1d0 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.592+0000 7f870d737640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.152:6792/0 conn(0x7f87080a3860 legacy=0x7f87081cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.592+0000 7f870e739640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.152:6789/0 conn(0x7f87080a4350 legacy=0x7f87081d35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 705325213 0 0) 0x7f86ec003520 con 0x7f870806c1d0 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f870809f200 con 0x7f870806c1d0 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8704003610 con 0x7f870806c1d0 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient(hunting): dump: 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.609 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245713688 0 0) 0x7f870809f730 con 0x7f87080a3860 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient(hunting): my global_id is 4606 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ec003520 con 0x7f87080a3860 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4219831193 0 0) 0x7f870809f200 con 0x7f870806c1d0 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.152:6792/0 conn(0x7f87080a3860 legacy=0x7f87081cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.152:6789/0 conn(0x7f87080a4350 legacy=0x7f87081d35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 monclient: found mon.e 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87081aac10 con 0x7f870806c1d0 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f871019c640 5 monclient: authenticate success, global_id 4553 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f871019c640 10 monclient: _renew_subs 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f871019c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6793/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87081aaef0 con 0x7f870806c1d0 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f871019c640 10 monclient: _renew_subs 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f871019c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87081a8e00 con 0x7f870806c1d0 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.8 v1:172.21.15.94:6793/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8704003be0 con 0x7f870806c1d0 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.8 v1:172.21.15.94:6793/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8704004090 con 0x7f870806c1d0 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fffff640 4 set_mon_vals no callback set 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.593+0000 7f86fe7fc640 10 monclient: dump: 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.594+0000 7f871019c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.594+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f86f0006580 con 0x7f86f0002820 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.594+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.8 v1:172.21.15.94:6793/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f87040041f0 con 0x7f870806c1d0 2024-09-16T10:31:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.594+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.8 v1:172.21.15.94:6793/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f8704045f10 con 0x7f870806c1d0 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.594+0000 7f870e739640 10 monclient: get_auth_request con 0x7f86ec038f80 auth_method 0 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f870d737640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.94:6791/0 conn(0x7f86f0002820 legacy=0x7f86f0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956826527 0 0) 0x7f86f0006580 con 0x7f86f0002820 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 10 monclient(hunting): my global_id is 4585 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f870809f200 con 0x7f86f0002820 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2661250726 0 0) 0x7f870809f200 con 0x7f86f0002820 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86f0006580 con 0x7f86f0002820 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f86f8001510 con 0x7f86f0002820 2024-09-16T10:31:49.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:49.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 10 monclient: dump: 2024-09-16T10:31:49.616 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.616 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.616 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.616 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.616 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.616 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.616 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.617 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4131018094 0 0) 0x7f86f0006580 con 0x7f86f0002820 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.596+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86f000a0d0 con 0x7f86f0002820 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.597+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f86f8001880 con 0x7f86f0002820 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.597+0000 7f86fe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.597+0000 7f86fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.597+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.94:6791/0 conn(0x7f86f0002820 legacy=0x7f86f0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.597+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.94:6791/0 conn(0x7f86f0002820 legacy=0x7f86f0004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.603+0000 7f871019c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.603+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f86f0009d90 con 0x7f86f00090b0 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f870d737640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.94:6791/0 conn(0x7f86f00090b0 legacy=0x7f86f0009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080699849 0 0) 0x7f86f0009d90 con 0x7f86f00090b0 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 10 monclient(hunting): my global_id is 4594 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f0006580 con 0x7f86f00090b0 2024-09-16T10:31:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3045560300 0 0) 0x7f86f0006580 con 0x7f86f00090b0 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86f0009d90 con 0x7f86f00090b0 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f86f8001a80 con 0x7f86f00090b0 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.605+0000 7f86fe7fc640 10 monclient: dump: 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.623 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 135399975 0 0) 0x7f86f0009d90 con 0x7f86f00090b0 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 --> v1:172.21.15.94:6791/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f86f000c280 con 0x7f86f00090b0 2024-09-16T10:31:49.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 <== mon.4 v1:172.21.15.94:6791/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7452 (unknown 0 0 2229213243) 0x7f86f80077d0 con 0x7f86f00090b0 2024-09-16T10:31:49.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f86fe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:31:49.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f86fe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:49.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.94:6791/0 conn(0x7f86f00090b0 legacy=0x7f86f0009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f86fe7fc640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.94:6791/0 conn(0x7f86f00090b0 legacy=0x7f86f0009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.625 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:49.625 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:49.625 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:49.625 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 18, 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.626 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 0, 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.627 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.628 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.628 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:31:49.628 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:31:49.628 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.628 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.628 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.628 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:31:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.630 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:31:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.632 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:31:49.633 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:49.634 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.635 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.638 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.639 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.640 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.641 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:31:49.641 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.641 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.641 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.641 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.641 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:31:49.642 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:49.643 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.644 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:31:49.646 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.152:6800/2686360781 conn(0x7f86ec038f80 legacy=0x7f86ec03b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f871019c640 10 monclient: shutdown 2024-09-16T10:31:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f871019c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 >> v1:172.21.15.94:6793/0 conn(0x7f870806c1d0 legacy=0x7f87081bd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f870e739640 1 -- 172.21.15.94:0/2416792159 reap_dead start 2024-09-16T10:31:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f870e739640 1 -- 172.21.15.94:0/2416792159 reap_dead start 2024-09-16T10:31:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 shutdown_connections 2024-09-16T10:31:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 >> 172.21.15.94:0/2416792159 conn(0x7f87080a5960 msgr2=0x7f87080a7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 shutdown_connections 2024-09-16T10:31:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.606+0000 7f871019c640 1 -- 172.21.15.94:0/2416792159 wait complete. 2024-09-16T10:31:49.649 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:31:49.649 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-16T10:31:49.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.744+0000 7fe2d7227640 10 monclient: get_monmap_and_config 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.744+0000 7fe2d7227640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.744+0000 7fe2d7227640 10 monclient: monmap: 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:49.745302+0000 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:49.745302+0000 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.746 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.747 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.747 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.747 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.747 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.747 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.747 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.747 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.747 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 1 Processor -- start 2024-09-16T10:31:49.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 1 -- start start 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 10 monclient: init 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 10 monclient: _add_conns ranks=[5,8,3,1,6,2,4,7,0] 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 10 monclient(hunting): picked mon.noname-h con 0x7fe2d00a3860 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 10 monclient(hunting): picked mon.noname-e con 0x7fe2d006c1d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 10 monclient(hunting): picked mon.noname-g con 0x7fe2d00a4350 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d0061ae0 con 0x7fe2d00a4350 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d00a0600 con 0x7fe2d00a3860 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d00a0b30 con 0x7fe2d006c1d0 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2d7227640 10 monclient(hunting): authenticate will time out at 877.085880s 2024-09-16T10:31:49.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2cffff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7fe2d006c1d0 0x7fe2d013b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:44006/0 (socket says 172.21.15.94:44006) 2024-09-16T10:31:49.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.745+0000 7fe2cffff640 1 -- 172.21.15.94:0/1032301034 learned_addr learned my addr 172.21.15.94:0/1032301034 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cffff640 1 -- 172.21.15.94:0/1032301034 >> v1:172.21.15.94:6793/0 conn(0x7fe2d006c1d0 legacy=0x7fe2d013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 -- 172.21.15.94:0/1032301034 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953859694 0 0) 0x7fe2d00a0b30 con 0x7fe2d006c1d0 2024-09-16T10:31:49.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient(hunting): my global_id is 4562 2024-09-16T10:31:49.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 -- 172.21.15.94:0/1032301034 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c0003520 con 0x7fe2d006c1d0 2024-09-16T10:31:49.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2d4fc3640 1 -- 172.21.15.94:0/1032301034 >> v1:172.21.15.152:6791/0 conn(0x7fe2d00a3860 legacy=0x7fe2d00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2d57c4640 1 -- 172.21.15.94:0/1032301034 >> v1:172.21.15.152:6790/0 conn(0x7fe2d00a4350 legacy=0x7fe2d013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 -- 172.21.15.94:0/1032301034 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207351889 0 0) 0x7fe2c0003520 con 0x7fe2d006c1d0 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 -- 172.21.15.94:0/1032301034 >> v1:172.21.15.152:6791/0 conn(0x7fe2d00a3860 legacy=0x7fe2d00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 -- 172.21.15.94:0/1032301034 >> v1:172.21.15.152:6790/0 conn(0x7fe2d00a4350 legacy=0x7fe2d013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 monclient: found mon.noname-e 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 -- 172.21.15.94:0/1032301034 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d00a0e10 con 0x7fe2d006c1d0 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 -- 172.21.15.94:0/1032301034 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe2bc001a60 con 0x7fe2d006c1d0 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: dump: 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.755 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2d7227640 5 monclient: authenticate success, global_id 4562 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2d7227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 -- 172.21.15.94:0/1032301034 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2bc001d70 con 0x7fe2d006c1d0 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 1 -- 172.21.15.94:0/1032301034 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe2bc004be0 con 0x7fe2d006c1d0 2024-09-16T10:31:49.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2d7227640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2d7227640 4 set_mon_vals no callback set 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.748+0000 7fe2cf7fe640 10 monclient: dump: 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.757 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.749+0000 7fe2d7227640 10 monclient: shutdown 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.749+0000 7fe2d7227640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.749+0000 7fe2d7227640 1 -- 172.21.15.94:0/1032301034 >> v1:172.21.15.94:6793/0 conn(0x7fe2d006c1d0 legacy=0x7fe2d013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.749+0000 7fe2d7227640 1 -- 172.21.15.94:0/1032301034 shutdown_connections 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.749+0000 7fe2d7227640 1 -- 172.21.15.94:0/1032301034 >> 172.21.15.94:0/1032301034 conn(0x7fe2d00a5960 msgr2=0x7fe2d00a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.749+0000 7fe2d7227640 1 -- 172.21.15.94:0/1032301034 shutdown_connections 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.749+0000 7fe2d7227640 1 -- 172.21.15.94:0/1032301034 wait complete. 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.749+0000 7fe2d7227640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.749+0000 7fe2d7227640 10 monclient: monmap: 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.759 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 1 Processor -- start 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 1 -- start start 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 10 monclient: init 2024-09-16T10:31:49.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 10 monclient: _add_conns ranks=[8,0,7,6,2,1,5,4,3] 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 10 monclient(hunting): picked mon.noname-e con 0x7fe2d006c1d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d00a3860 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 10 monclient(hunting): picked mon.noname-i con 0x7fe2d00a4350 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d01cbb30 con 0x7fe2d00a3860 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d009f200 con 0x7fe2d00a4350 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d006ba10 con 0x7fe2d006c1d0 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d7227640 10 monclient(hunting): authenticate will time out at 877.090725s 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d4fc3640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7fe2d006c1d0 0x7fe2d01bb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:44020/0 (socket says 172.21.15.94:44020) 2024-09-16T10:31:49.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.750+0000 7fe2d4fc3640 1 -- 172.21.15.94:0/185064579 learned_addr learned my addr 172.21.15.94:0/185064579 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.752+0000 7fe2d4fc3640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6793/0 conn(0x7fe2d006c1d0 legacy=0x7fe2d01bb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.752+0000 7fe2cffff640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6789/0 conn(0x7fe2d00a3860 legacy=0x7fe2d01cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479221408 0 0) 0x7fe2d006ba10 con 0x7fe2d006c1d0 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): my global_id is 4571 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b4003520 con 0x7fe2d006c1d0 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938697713 0 0) 0x7fe2d01cbb30 con 0x7fe2d00a3860 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): my global_id is 4554 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d006ba10 con 0x7fe2d00a3860 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2d57c4640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.152:6792/0 conn(0x7fe2d00a4350 legacy=0x7fe2d01d3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2375003377 0 0) 0x7fe2b4003520 con 0x7fe2d006c1d0 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2d01cbb30 con 0x7fe2d006c1d0 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe2c4003610 con 0x7fe2d006c1d0 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): dump: 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.767 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.768 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 903455183 0 0) 0x7fe2d006ba10 con 0x7fe2d00a3860 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b4003520 con 0x7fe2d00a3860 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe2bc004b20 con 0x7fe2d00a3860 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): dump: 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504453334 0 0) 0x7fe2d009f200 con 0x7fe2d00a4350 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient(hunting): my global_id is 4615 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d006ba10 con 0x7fe2d00a4350 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3270395233 0 0) 0x7fe2d01cbb30 con 0x7fe2d006c1d0 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.152:6792/0 conn(0x7fe2d00a4350 legacy=0x7fe2d01d3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6789/0 conn(0x7fe2d00a3860 legacy=0x7fe2d01cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 monclient: found mon.e 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d01a90b0 con 0x7fe2d006c1d0 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2d7227640 5 monclient: authenticate success, global_id 4571 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 341651078 0 0) 0x7fe2b4003520 con 0x7fe2d00a3860 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2d7227640 10 monclient: _renew_subs 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2d7227640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6793/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d01a8810 con 0x7fe2d006c1d0 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2d7227640 10 monclient: _renew_subs 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2d7227640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6793/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d01cae70 con 0x7fe2d006c1d0 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.8 v1:172.21.15.94:6793/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2c4003b90 con 0x7fe2d006c1d0 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.8 v1:172.21.15.94:6793/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe2c4004040 con 0x7fe2d006c1d0 2024-09-16T10:31:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2ceffd640 4 set_mon_vals no callback set 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.753+0000 7fe2cd7fa640 10 monclient: dump: 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.773 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.754+0000 7fe2d7227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.754+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe2b0006580 con 0x7fe2b0002820 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.754+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.8 v1:172.21.15.94:6793/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fe2c40041a0 con 0x7fe2d006c1d0 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.754+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.8 v1:172.21.15.94:6793/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7fe2c4045f80 con 0x7fe2d006c1d0 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.755+0000 7fe2d57c4640 10 monclient: get_auth_request con 0x7fe2b4039840 auth_method 0 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.756+0000 7fe2cffff640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6792/0 conn(0x7fe2b0002820 legacy=0x7fe2b0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.756+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834281388 0 0) 0x7fe2b0006580 con 0x7fe2b0002820 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.756+0000 7fe2cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.756+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.756+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.756+0000 7fe2cd7fa640 10 monclient(hunting): my global_id is 94119 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.756+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b4003520 con 0x7fe2b0002820 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 953525614 0 0) 0x7fe2b4003520 con 0x7fe2b0002820 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b0006580 con 0x7fe2b0002820 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe2bc001580 con 0x7fe2b0002820 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 10 monclient: dump: 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.775 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.776 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3020637342 0 0) 0x7fe2b0006580 con 0x7fe2b0002820 2024-09-16T10:31:49.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2b000a0d0 con 0x7fe2b0002820 2024-09-16T10:31:49.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe2bc007720 con 0x7fe2b0002820 2024-09-16T10:31:49.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:49.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6792/0 conn(0x7fe2b0002820 legacy=0x7fe2b0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.757+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6792/0 conn(0x7fe2b0002820 legacy=0x7fe2b0004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.763+0000 7fe2d7227640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:31:49.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.763+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe2b0009d90 con 0x7fe2b00090b0 2024-09-16T10:31:49.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.765+0000 7fe2cffff640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6792/0 conn(0x7fe2b00090b0 legacy=0x7fe2b0009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583901399 0 0) 0x7fe2b0009d90 con 0x7fe2b00090b0 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 10 monclient(hunting): my global_id is 94128 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b0006580 con 0x7fe2b00090b0 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1851640495 0 0) 0x7fe2b0006580 con 0x7fe2b00090b0 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b0009d90 con 0x7fe2b00090b0 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe2bc005e30 con 0x7fe2b00090b0 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 10 monclient: dump: 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3630069878 0 0) 0x7fe2b0009d90 con 0x7fe2b00090b0 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 --> v1:172.21.15.94:6792/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe2b000c280 con 0x7fe2b00090b0 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 <== mon.6 v1:172.21.15.94:6792/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7452 (unknown 0 0 1306589261) 0x7fe2bc0073b0 con 0x7fe2b00090b0 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6792/0 conn(0x7fe2b00090b0 legacy=0x7fe2b0009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2cd7fa640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6792/0 conn(0x7fe2b00090b0 legacy=0x7fe2b0009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 18, 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:31:49.782 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 0, 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.783 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.784 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:31:49.785 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.787 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:31:49.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.788 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:31:49.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.788 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:49.788 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:49.788 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.788 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.788 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.790 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:31:49.791 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:49.792 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.795 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.796 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.798 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:49.799 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.800 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.801 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:31:49.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.802 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.802 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.802 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.802 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.802 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:31:49.802 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:49.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.152:6800/2686360781 conn(0x7fe2b4039840 legacy=0x7fe2b403bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2d7227640 10 monclient: shutdown 2024-09-16T10:31:49.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2d7227640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 >> v1:172.21.15.94:6793/0 conn(0x7fe2d006c1d0 legacy=0x7fe2d01bb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2d57c4640 1 -- 172.21.15.94:0/185064579 reap_dead start 2024-09-16T10:31:49.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.766+0000 7fe2d57c4640 1 -- 172.21.15.94:0/185064579 reap_dead start 2024-09-16T10:31:49.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.767+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 shutdown_connections 2024-09-16T10:31:49.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.767+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 >> 172.21.15.94:0/185064579 conn(0x7fe2d00a5960 msgr2=0x7fe2d00a7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.767+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 shutdown_connections 2024-09-16T10:31:49.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.767+0000 7fe2d7227640 1 -- 172.21.15.94:0/185064579 wait complete. 2024-09-16T10:31:49.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:31:49.805 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-16T10:31:49.805 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-16T10:31:49.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.901+0000 7f88bc041640 10 monclient: get_monmap_and_config 2024-09-16T10:31:49.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.901+0000 7f88bc041640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.901+0000 7f88bc041640 10 monclient: monmap: 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:49.902010+0000 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:49.902010+0000 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.903 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.904 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.904 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.904 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.904 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.901+0000 7f88bc041640 1 Processor -- start 2024-09-16T10:31:49.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.901+0000 7f88bc041640 1 -- start start 2024-09-16T10:31:49.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.901+0000 7f88bc041640 10 monclient: init 2024-09-16T10:31:49.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 10 monclient: _add_conns ranks=[5,3,8,6,2,4,0,7,1] 2024-09-16T10:31:49.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 10 monclient(hunting): picked mon.noname-h con 0x7f88b40a3860 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:49.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 10 monclient(hunting): picked mon.noname-g con 0x7f88b406c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:49.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 10 monclient(hunting): picked mon.noname-e con 0x7f88b40a4350 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:49.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b4061ae0 con 0x7f88b406c1d0 2024-09-16T10:31:49.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40a0600 con 0x7f88b40a3860 2024-09-16T10:31:49.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40a0b30 con 0x7f88b40a4350 2024-09-16T10:31:49.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88bc041640 10 monclient(hunting): authenticate will time out at 877.242467s 2024-09-16T10:31:49.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88ba5de640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f88b40a4350 0x7f88b413ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:44024/0 (socket says 172.21.15.94:44024) 2024-09-16T10:31:49.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.902+0000 7f88ba5de640 1 -- 172.21.15.94:0/917363262 learned_addr learned my addr 172.21.15.94:0/917363262 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.904+0000 7f88ba5de640 1 -- 172.21.15.94:0/917363262 >> v1:172.21.15.94:6793/0 conn(0x7f88b40a4350 legacy=0x7f88b413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 -- 172.21.15.94:0/917363262 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252718796 0 0) 0x7f88b40a0b30 con 0x7f88b40a4350 2024-09-16T10:31:49.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient(hunting): my global_id is 4580 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 -- 172.21.15.94:0/917363262 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a0003520 con 0x7f88b40a4350 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b95dc640 1 -- 172.21.15.94:0/917363262 >> v1:172.21.15.152:6790/0 conn(0x7f88b406c1d0 legacy=0x7f88b413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b9ddd640 1 -- 172.21.15.94:0/917363262 >> v1:172.21.15.152:6791/0 conn(0x7f88b40a3860 legacy=0x7f88b40a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 -- 172.21.15.94:0/917363262 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 634100009 0 0) 0x7f88a0003520 con 0x7f88b40a4350 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 -- 172.21.15.94:0/917363262 >> v1:172.21.15.152:6791/0 conn(0x7f88b40a3860 legacy=0x7f88b40a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 -- 172.21.15.94:0/917363262 >> v1:172.21.15.152:6790/0 conn(0x7f88b406c1d0 legacy=0x7f88b413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 monclient: found mon.noname-e 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 -- 172.21.15.94:0/917363262 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40a0e10 con 0x7f88b40a4350 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 -- 172.21.15.94:0/917363262 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f88ac001de0 con 0x7f88b40a4350 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 5 monclient: authenticate success, global_id 4580 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: dump: 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.911 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 -- 172.21.15.94:0/917363262 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88ac0020f0 con 0x7f88b40a4350 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 1 -- 172.21.15.94:0/917363262 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f88ac004f60 con 0x7f88b40a4350 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88b8ddb640 10 monclient: dump: 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.913 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 4 set_mon_vals no callback set 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 10 monclient: shutdown 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 1 -- 172.21.15.94:0/917363262 >> v1:172.21.15.94:6793/0 conn(0x7f88b40a4350 legacy=0x7f88b413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 1 -- 172.21.15.94:0/917363262 shutdown_connections 2024-09-16T10:31:49.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.905+0000 7f88bc041640 1 -- 172.21.15.94:0/917363262 >> 172.21.15.94:0/917363262 conn(0x7f88b40a5960 msgr2=0x7f88b40a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.906+0000 7f88bc041640 1 -- 172.21.15.94:0/917363262 shutdown_connections 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.906+0000 7f88bc041640 1 -- 172.21.15.94:0/917363262 wait complete. 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.906+0000 7f88bc041640 10 monclient: build_initial_monmap 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.906+0000 7f88bc041640 10 monclient: monmap: 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:49.915 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.906+0000 7f88bc041640 1 Processor -- start 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.906+0000 7f88bc041640 1 -- start start 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.906+0000 7f88bc041640 10 monclient: init 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 10 monclient: _add_conns ranks=[3,4,7,6,1,5,2,8,0] 2024-09-16T10:31:49.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 10 monclient(hunting): picked mon.noname-g con 0x7f88b406c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 10 monclient(hunting): picked mon.noname-c con 0x7f88b40a3860 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 10 monclient(hunting): picked mon.noname-i con 0x7f88b40a4350 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b409f300 con 0x7f88b406c1d0 2024-09-16T10:31:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b406ba50 con 0x7f88b40a3860 2024-09-16T10:31:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b41a9090 con 0x7f88b40a4350 2024-09-16T10:31:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88bc041640 10 monclient(hunting): authenticate will time out at 877.247371s 2024-09-16T10:31:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88b95dc640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f88b40a3860 0x7f88b41bb6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33986/0 (socket says 172.21.15.94:33986) 2024-09-16T10:31:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.907+0000 7f88b95dc640 1 -- 172.21.15.94:0/1855768238 learned_addr learned my addr 172.21.15.94:0/1855768238 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:49.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.909+0000 7f88b95dc640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.94:6791/0 conn(0x7f88b40a3860 legacy=0x7f88b41bb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.909+0000 7f88b9ddd640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.152:6790/0 conn(0x7f88b406c1d0 legacy=0x7f88b41cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.909+0000 7f88ba5de640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.152:6792/0 conn(0x7f88b40a4350 legacy=0x7f88b41d2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.909+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401760506 0 0) 0x7f88b406ba50 con 0x7f88b40a3860 2024-09-16T10:31:49.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.909+0000 7f88a67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.909+0000 7f88a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.909+0000 7f88a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.909+0000 7f88a67fc640 10 monclient(hunting): my global_id is 4603 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.909+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8898003520 con 0x7f88b40a3860 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 339339440 0 0) 0x7f8898003520 con 0x7f88b40a3860 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88b406ba50 con 0x7f88b40a3860 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f88a8003610 con 0x7f88b40a3860 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): dump: 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.922 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.923 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129434312 0 0) 0x7f88b41a9090 con 0x7f88b40a4350 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): my global_id is 4624 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8898003520 con 0x7f88b40a4350 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866131937 0 0) 0x7f88b409f300 con 0x7f88b406c1d0 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient(hunting): my global_id is 4782 2024-09-16T10:31:49.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b41a9090 con 0x7f88b406c1d0 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1565911731 0 0) 0x7f88b406ba50 con 0x7f88b40a3860 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.152:6792/0 conn(0x7f88b40a4350 legacy=0x7f88b41d2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.152:6790/0 conn(0x7f88b406c1d0 legacy=0x7f88b41cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 monclient: found mon.c 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b409f850 con 0x7f88b40a3860 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88bc041640 5 monclient: authenticate success, global_id 4603 2024-09-16T10:31:49.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88bc041640 10 monclient: _renew_subs 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88bc041640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b41a8db0 con 0x7f88b40a3860 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88bc041640 10 monclient: _renew_subs 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88bc041640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b41a7650 con 0x7f88b40a3860 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88a8003c50 con 0x7f88b40a3860 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f88a8004100 con 0x7f88b40a3860 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a7fff640 4 set_mon_vals no callback set 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.910+0000 7f88a67fc640 10 monclient: dump: 2024-09-16T10:31:49.926 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.911+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f88a80042e0 con 0x7f88b40a3860 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.911+0000 7f88bc041640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.911+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f889c006580 con 0x7f889c002820 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.911+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f88a8046150 con 0x7f88b40a3860 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.911+0000 7f88ba5de640 10 monclient: get_auth_request con 0x7f88980391d0 auth_method 0 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88b9ddd640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.94:6793/0 conn(0x7f889c002820 legacy=0x7f889c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317407380 0 0) 0x7f889c006580 con 0x7f889c002820 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 10 monclient(hunting): my global_id is 4589 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b406ba50 con 0x7f889c002820 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1289564356 0 0) 0x7f88b406ba50 con 0x7f889c002820 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f889c006580 con 0x7f889c002820 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f88b0001b70 con 0x7f889c002820 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 10 monclient: dump: 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2221804175 0 0) 0x7f889c006580 con 0x7f889c002820 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.913+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f889c00a0d0 con 0x7f889c002820 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.914+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f88b00057c0 con 0x7f889c002820 2024-09-16T10:31:49.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.914+0000 7f88a67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:49.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.914+0000 7f88a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:49.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.914+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.94:6793/0 conn(0x7f889c002820 legacy=0x7f889c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.914+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.94:6793/0 conn(0x7f889c002820 legacy=0x7f889c004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.920+0000 7f88bc041640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:31:49.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.920+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f889c009d70 con 0x7f889c009090 2024-09-16T10:31:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88b9ddd640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.94:6793/0 conn(0x7f889c009090 legacy=0x7f889c009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815404478 0 0) 0x7f889c009d70 con 0x7f889c009090 2024-09-16T10:31:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 10 monclient(hunting): my global_id is 4598 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f889c006580 con 0x7f889c009090 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4245877780 0 0) 0x7f889c006580 con 0x7f889c009090 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f889c009d70 con 0x7f889c009090 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f88b0003db0 con 0x7f889c009090 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 10 monclient: dump: 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:49.934 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 386845621 0 0) 0x7f889c009d70 con 0x7f889c009090 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.922+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 --> v1:172.21.15.94:6793/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f889c00c280 con 0x7f889c009090 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 <== mon.8 v1:172.21.15.94:6793/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7618 (unknown 0 0 1215559845) 0x7f88b00042f0 con 0x7f889c009090 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88a67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88a67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.94:6793/0 conn(0x7f889c009090 legacy=0x7f889c009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88a67fc640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.94:6793/0 conn(0x7f889c009090 legacy=0x7f889c009520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:49.935 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 18, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:31:49.936 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 1, 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.937 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.938 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:49.939 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:49.940 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:49.941 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.942 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.944 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.945 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.946 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.947 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.948 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.949 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:49.951 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.952 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:49.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:31:49.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.955 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:31:49.956 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:49.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.152:6800/2686360781 conn(0x7f88980391d0 legacy=0x7f889803b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88bc041640 10 monclient: shutdown 2024-09-16T10:31:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88bc041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 >> v1:172.21.15.94:6791/0 conn(0x7f88b40a3860 legacy=0x7f88b41bb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88ba5de640 1 -- 172.21.15.94:0/1855768238 reap_dead start 2024-09-16T10:31:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88ba5de640 1 -- 172.21.15.94:0/1855768238 reap_dead start 2024-09-16T10:31:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 shutdown_connections 2024-09-16T10:31:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 >> 172.21.15.94:0/1855768238 conn(0x7f88b40a5960 msgr2=0x7f88b40a7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 shutdown_connections 2024-09-16T10:31:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:49.923+0000 7f88bc041640 1 -- 172.21.15.94:0/1855768238 wait complete. 2024-09-16T10:31:49.959 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:31:49.959 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-16T10:31:49.959 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-16T10:31:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.056+0000 7fc51022b640 10 monclient: get_monmap_and_config 2024-09-16T10:31:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.056+0000 7fc51022b640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.056+0000 7fc51022b640 10 monclient: monmap: 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:50.056995+0000 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:50.056995+0000 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.059 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.059 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.059 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.056+0000 7fc51022b640 1 Processor -- start 2024-09-16T10:31:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.056+0000 7fc51022b640 1 -- start start 2024-09-16T10:31:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.056+0000 7fc51022b640 10 monclient: init 2024-09-16T10:31:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 10 monclient: _add_conns ranks=[1,5,6,8,2,7,3,0,4] 2024-09-16T10:31:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 10 monclient(hunting): picked mon.noname-f con 0x7fc5080a38d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 10 monclient(hunting): picked mon.noname-h con 0x7fc50806c670 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 10 monclient(hunting): picked mon.noname-d con 0x7fc5080a43c0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc508061ae0 con 0x7fc5080a38d0 2024-09-16T10:31:50.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5080a04d0 con 0x7fc50806c670 2024-09-16T10:31:50.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5080a3250 con 0x7fc5080a43c0 2024-09-16T10:31:50.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc51022b640 10 monclient(hunting): authenticate will time out at 877.397469s 2024-09-16T10:31:50.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc50e7c8640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fc5080a43c0 0x7fc50813f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:48974/0 (socket says 172.21.15.94:48974) 2024-09-16T10:31:50.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.057+0000 7fc50e7c8640 1 -- 172.21.15.94:0/192220618 learned_addr learned my addr 172.21.15.94:0/192220618 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.059+0000 7fc50e7c8640 1 -- 172.21.15.94:0/192220618 >> v1:172.21.15.94:6792/0 conn(0x7fc5080a43c0 legacy=0x7fc50813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.059+0000 7fc50dfc7640 1 -- 172.21.15.94:0/192220618 >> v1:172.21.15.152:6789/0 conn(0x7fc5080a38d0 legacy=0x7fc5080a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.059+0000 7fc50d7c6640 1 -- 172.21.15.94:0/192220618 >> v1:172.21.15.152:6791/0 conn(0x7fc50806c670 legacy=0x7fc50813bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078922898 0 0) 0x7fc5080a3250 con 0x7fc5080a43c0 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): my global_id is 94137 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f0003520 con 0x7fc5080a43c0 2024-09-16T10:31:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296909979 0 0) 0x7fc508061ae0 con 0x7fc5080a38d0 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): my global_id is 4537 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5080a3250 con 0x7fc5080a38d0 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120340810 0 0) 0x7fc5080a04d0 con 0x7fc50806c670 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient(hunting): my global_id is 4622 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc508061ae0 con 0x7fc50806c670 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3809387548 0 0) 0x7fc4f0003520 con 0x7fc5080a43c0 2024-09-16T10:31:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 >> v1:172.21.15.152:6791/0 conn(0x7fc50806c670 legacy=0x7fc50813bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 >> v1:172.21.15.152:6789/0 conn(0x7fc5080a38d0 legacy=0x7fc5080a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 monclient: found mon.noname-d 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080a0a00 con 0x7fc5080a43c0 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 5 monclient: authenticate success, global_id 94137 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4f8001ea0 con 0x7fc5080a43c0 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: dump: 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4f8004c20 con 0x7fc5080a43c0 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 1 -- 172.21.15.94:0/192220618 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4f80050d0 con 0x7fc5080a43c0 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 4 set_mon_vals no callback set 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:50.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc50cfc5640 10 monclient: dump: 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.070 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 10 monclient: shutdown 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 1 -- 172.21.15.94:0/192220618 >> v1:172.21.15.94:6792/0 conn(0x7fc5080a43c0 legacy=0x7fc50813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 1 -- 172.21.15.94:0/192220618 shutdown_connections 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 1 -- 172.21.15.94:0/192220618 >> 172.21.15.94:0/192220618 conn(0x7fc5080a59d0 msgr2=0x7fc5080a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 1 -- 172.21.15.94:0/192220618 shutdown_connections 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.060+0000 7fc51022b640 1 -- 172.21.15.94:0/192220618 wait complete. 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.061+0000 7fc51022b640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.061+0000 7fc51022b640 10 monclient: monmap: 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:50.071 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.072 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.061+0000 7fc51022b640 1 Processor -- start 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.061+0000 7fc51022b640 1 -- start start 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.061+0000 7fc51022b640 10 monclient: init 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 10 monclient: _add_conns ranks=[7,6,2,1,5,0,8,4,3] 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 10 monclient(hunting): picked mon.noname-i con 0x7fc50806c670 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 10 monclient(hunting): picked mon.noname-d con 0x7fc5080a38d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 10 monclient(hunting): picked mon.noname-b con 0x7fc5080a43c0 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50806bb00 con 0x7fc5080a43c0 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50809f350 con 0x7fc5080a38d0 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc508125cd0 con 0x7fc50806c670 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc51022b640 10 monclient(hunting): authenticate will time out at 877.402443s 2024-09-16T10:31:50.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc50d7c6640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fc5080a38d0 0x7fc5081cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:48984/0 (socket says 172.21.15.94:48984) 2024-09-16T10:31:50.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.062+0000 7fc50d7c6640 1 -- 172.21.15.94:0/3045011329 learned_addr learned my addr 172.21.15.94:0/3045011329 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc50e7c8640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.94:6790/0 conn(0x7fc5080a43c0 legacy=0x7fc5081d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc50d7c6640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.94:6792/0 conn(0x7fc5080a38d0 legacy=0x7fc5081cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc50dfc7640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.152:6792/0 conn(0x7fc50806c670 legacy=0x7fc508136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347526099 0 0) 0x7fc50806bb00 con 0x7fc5080a43c0 2024-09-16T10:31:50.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): my global_id is 4565 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f4003520 con 0x7fc5080a43c0 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152926328 0 0) 0x7fc508061ae0 con 0x7fc5080a38d0 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): my global_id is 94146 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc50806bb00 con 0x7fc5080a38d0 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725465857 0 0) 0x7fc508125cd0 con 0x7fc50806c670 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 10 monclient(hunting): my global_id is 4633 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc508061ae0 con 0x7fc50806c670 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.064+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359829838 0 0) 0x7fc4f4003520 con 0x7fc5080a43c0 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc508125cd0 con 0x7fc5080a43c0 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4f8001e60 con 0x7fc5080a43c0 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): dump: 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.081 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1050149154 0 0) 0x7fc50806bb00 con 0x7fc5080a38d0 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4f4003520 con 0x7fc5080a38d0 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4fc003610 con 0x7fc5080a38d0 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): dump: 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4131958049 0 0) 0x7fc508061ae0 con 0x7fc50806c670 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc50806bb00 con 0x7fc50806c670 2024-09-16T10:31:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc500001d70 con 0x7fc50806c670 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient(hunting): dump: 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1508304951 0 0) 0x7fc508125cd0 con 0x7fc5080a43c0 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.152:6792/0 conn(0x7fc50806c670 legacy=0x7fc508136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.94:6792/0 conn(0x7fc5080a38d0 legacy=0x7fc5081cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 monclient: found mon.b 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50809efb0 con 0x7fc5080a43c0 2024-09-16T10:31:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc51022b640 5 monclient: authenticate success, global_id 4565 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc51022b640 10 monclient: _renew_subs 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc51022b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc508127240 con 0x7fc5080a43c0 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc51022b640 10 monclient: _renew_subs 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc51022b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc508124ee0 con 0x7fc5080a43c0 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4f8005550 con 0x7fc5080a43c0 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc4f8005a00 con 0x7fc5080a43c0 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc507fff640 4 set_mon_vals no callback set 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc5067fc640 10 monclient: dump: 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.065+0000 7fc51022b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.066+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc4e8006580 con 0x7fc4e8002820 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.066+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fc4f8010a60 con 0x7fc5080a43c0 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.066+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7fc4f80463c0 con 0x7fc5080a43c0 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.066+0000 7fc50d7c6640 10 monclient: get_auth_request con 0x7fc4f4039ce0 auth_method 0 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc50dfc7640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.152:6789/0 conn(0x7fc4e8002820 legacy=0x7fc4e8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941581825 0 0) 0x7fc4e8006580 con 0x7fc4e8002820 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 10 monclient(hunting): my global_id is 4546 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc508125cd0 con 0x7fc4e8002820 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2610851375 0 0) 0x7fc508125cd0 con 0x7fc4e8002820 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4e8006580 con 0x7fc4e8002820 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc500003c50 con 0x7fc4e8002820 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.068+0000 7fc5067fc640 10 monclient: dump: 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.069+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 203210591 0 0) 0x7fc4e8006580 con 0x7fc4e8002820 2024-09-16T10:31:50.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.069+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4e800a0d0 con 0x7fc4e8002820 2024-09-16T10:31:50.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.069+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc500004220 con 0x7fc4e8002820 2024-09-16T10:31:50.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.069+0000 7fc5067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.069+0000 7fc5067fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:50.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.069+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.152:6789/0 conn(0x7fc4e8002820 legacy=0x7fc4e8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.069+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.152:6789/0 conn(0x7fc4e8002820 legacy=0x7fc4e8004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:50.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.075+0000 7fc51022b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:31:50.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.075+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc4e8009d90 con 0x7fc4e80090b0 2024-09-16T10:31:50.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.078+0000 7fc50dfc7640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.152:6789/0 conn(0x7fc4e80090b0 legacy=0x7fc4e8009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.078+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189257706 0 0) 0x7fc4e8009d90 con 0x7fc4e80090b0 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.078+0000 7fc5067fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.078+0000 7fc5067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.078+0000 7fc5067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.078+0000 7fc5067fc640 10 monclient(hunting): my global_id is 4555 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.078+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e8006580 con 0x7fc4e80090b0 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.079+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3453895735 0 0) 0x7fc4e8006580 con 0x7fc4e80090b0 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.079+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4e8009d90 con 0x7fc4e80090b0 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.079+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fc500005b80 con 0x7fc4e80090b0 2024-09-16T10:31:50.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.079+0000 7fc5067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.079+0000 7fc5067fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.079+0000 7fc5067fc640 10 monclient: dump: 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.094 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.079+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 932026296 0 0) 0x7fc4e8009d90 con 0x7fc4e80090b0 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.079+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 --> v1:172.21.15.152:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc4e800c280 con 0x7fc4e80090b0 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 <== mon.1 v1:172.21.15.152:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7452 (unknown 0 0 2543805158) 0x7fc500007e90 con 0x7fc4e80090b0 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc5067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc5067fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.152:6789/0 conn(0x7fc4e80090b0 legacy=0x7fc4e8009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc5067fc640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.152:6789/0 conn(0x7fc4e80090b0 legacy=0x7fc4e8009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 18, 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:31:50.096 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 1, 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.097 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.098 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:31:50.099 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.100 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:31:50.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.102 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.103 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.104 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.105 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.107 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.108 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:50.109 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.110 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.112 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:31:50.114 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:50.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:31:50.116 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:50.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.152:6800/2686360781 conn(0x7fc4f4039ce0 legacy=0x7fc4f403c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc51022b640 10 monclient: shutdown 2024-09-16T10:31:50.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc51022b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 >> v1:172.21.15.94:6790/0 conn(0x7fc5080a43c0 legacy=0x7fc5081d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc50e7c8640 1 -- 172.21.15.94:0/3045011329 reap_dead start 2024-09-16T10:31:50.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc50e7c8640 1 -- 172.21.15.94:0/3045011329 reap_dead start 2024-09-16T10:31:50.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 shutdown_connections 2024-09-16T10:31:50.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 >> 172.21.15.94:0/3045011329 conn(0x7fc5080a59d0 msgr2=0x7fc50809ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 shutdown_connections 2024-09-16T10:31:50.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.080+0000 7fc51022b640 1 -- 172.21.15.94:0/3045011329 wait complete. 2024-09-16T10:31:50.119 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:31:50.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-16T10:31:50.119 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-16T10:31:50.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.219+0000 7fad7cae9640 10 monclient: get_monmap_and_config 2024-09-16T10:31:50.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.219+0000 7fad7cae9640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.219+0000 7fad7cae9640 10 monclient: monmap: 2024-09-16T10:31:50.221 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.221 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.221 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:50.220748+0000 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:50.220748+0000 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.222 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.220+0000 7fad7cae9640 1 Processor -- start 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.220+0000 7fad7cae9640 1 -- start start 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.220+0000 7fad7cae9640 10 monclient: init 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 10 monclient: _add_conns ranks=[5,4,3,7,8,0,2,6,1] 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 10 monclient(hunting): picked mon.noname-h con 0x7fad780a3860 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 10 monclient(hunting): picked mon.noname-c con 0x7fad7806c1d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 10 monclient(hunting): picked mon.noname-g con 0x7fad780a4350 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad78061ae0 con 0x7fad780a4350 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780a0600 con 0x7fad7806c1d0 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780a0b30 con 0x7fad780a3860 2024-09-16T10:31:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad7cae9640 10 monclient(hunting): authenticate will time out at 877.561130s 2024-09-16T10:31:50.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad6dd9b640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fad7806c1d0 0x7fad7813b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:33990/0 (socket says 172.21.15.94:33990) 2024-09-16T10:31:50.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.221+0000 7fad6dd9b640 1 -- 172.21.15.94:0/2677378151 learned_addr learned my addr 172.21.15.94:0/2677378151 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad6dd9b640 1 -- 172.21.15.94:0/2677378151 >> v1:172.21.15.94:6791/0 conn(0x7fad7806c1d0 legacy=0x7fad7813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 -- 172.21.15.94:0/2677378151 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394787126 0 0) 0x7fad780a0600 con 0x7fad7806c1d0 2024-09-16T10:31:50.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient(hunting): my global_id is 4612 2024-09-16T10:31:50.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 -- 172.21.15.94:0/2677378151 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad64003520 con 0x7fad7806c1d0 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad7659c640 1 -- 172.21.15.94:0/2677378151 >> v1:172.21.15.152:6791/0 conn(0x7fad780a3860 legacy=0x7fad780a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad76d9d640 1 -- 172.21.15.94:0/2677378151 >> v1:172.21.15.152:6790/0 conn(0x7fad780a4350 legacy=0x7fad7813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 -- 172.21.15.94:0/2677378151 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3837388340 0 0) 0x7fad64003520 con 0x7fad7806c1d0 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 -- 172.21.15.94:0/2677378151 >> v1:172.21.15.152:6791/0 conn(0x7fad780a3860 legacy=0x7fad780a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 -- 172.21.15.94:0/2677378151 >> v1:172.21.15.152:6790/0 conn(0x7fad780a4350 legacy=0x7fad7813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 monclient: found mon.noname-c 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 -- 172.21.15.94:0/2677378151 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad780a0e10 con 0x7fad7806c1d0 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 -- 172.21.15.94:0/2677378151 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fad60001a60 con 0x7fad7806c1d0 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad7cae9640 5 monclient: authenticate success, global_id 4612 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad7cae9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: dump: 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.230 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad7cae9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 -- 172.21.15.94:0/2677378151 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad60001d70 con 0x7fad7806c1d0 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 1 -- 172.21.15.94:0/2677378151 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fad60004be0 con 0x7fad7806c1d0 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad7cae9640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:50.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad7cae9640 4 set_mon_vals no callback set 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.223+0000 7fad75d9b640 10 monclient: dump: 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.224+0000 7fad7cae9640 10 monclient: shutdown 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.224+0000 7fad7cae9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.224+0000 7fad7cae9640 1 -- 172.21.15.94:0/2677378151 >> v1:172.21.15.94:6791/0 conn(0x7fad7806c1d0 legacy=0x7fad7813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.224+0000 7fad7cae9640 1 -- 172.21.15.94:0/2677378151 shutdown_connections 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.224+0000 7fad7cae9640 1 -- 172.21.15.94:0/2677378151 >> 172.21.15.94:0/2677378151 conn(0x7fad780a5960 msgr2=0x7fad780a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.224+0000 7fad7cae9640 1 -- 172.21.15.94:0/2677378151 shutdown_connections 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.224+0000 7fad7cae9640 1 -- 172.21.15.94:0/2677378151 wait complete. 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.225+0000 7fad7cae9640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.225+0000 7fad7cae9640 10 monclient: monmap: 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.225+0000 7fad7cae9640 1 Processor -- start 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.225+0000 7fad7cae9640 1 -- start start 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.225+0000 7fad7cae9640 10 monclient: init 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 10 monclient: _add_conns ranks=[6,3,8,2,5,1,0,4,7] 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 10 monclient(hunting): picked mon.noname-d con 0x7fad7806c1d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 10 monclient(hunting): picked mon.noname-g con 0x7fad780a3860 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 10 monclient(hunting): picked mon.noname-e con 0x7fad780a4350 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7809f7c0 con 0x7fad780a3860 2024-09-16T10:31:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7809f270 con 0x7fad7806c1d0 2024-09-16T10:31:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad781abfe0 con 0x7fad780a4350 2024-09-16T10:31:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7cae9640 10 monclient(hunting): authenticate will time out at 877.566203s 2024-09-16T10:31:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7659c640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fad7806c1d0 0x7fad781be620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:48998/0 (socket says 172.21.15.94:48998) 2024-09-16T10:31:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.226+0000 7fad7659c640 1 -- 172.21.15.94:0/2389373598 learned_addr learned my addr 172.21.15.94:0/2389373598 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad76d9d640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.94:6793/0 conn(0x7fad780a4350 legacy=0x7fad781d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad7659c640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.94:6792/0 conn(0x7fad7806c1d0 legacy=0x7fad781be620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927710755 0 0) 0x7fad781abfe0 con 0x7fad780a4350 2024-09-16T10:31:50.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): my global_id is 4607 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad58003520 con 0x7fad780a4350 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400267703 0 0) 0x7fad7809f270 con 0x7fad7806c1d0 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): my global_id is 94155 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad781abfe0 con 0x7fad7806c1d0 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6dd9b640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.152:6790/0 conn(0x7fad780a3860 legacy=0x7fad781cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3704833644 0 0) 0x7fad781abfe0 con 0x7fad7806c1d0 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad7809f270 con 0x7fad7806c1d0 2024-09-16T10:31:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fad68001d30 con 0x7fad7806c1d0 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): dump: 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.243 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582536401 0 0) 0x7fad7809f7c0 con 0x7fad780a3860 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): my global_id is 4800 2024-09-16T10:31:50.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad781abfe0 con 0x7fad780a3860 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2964611046 0 0) 0x7fad58003520 con 0x7fad780a4350 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad7809f7c0 con 0x7fad780a4350 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fad70001e70 con 0x7fad780a4350 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 10 monclient(hunting): dump: 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.245 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2865748313 0 0) 0x7fad7809f270 con 0x7fad7806c1d0 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.228+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.94:6793/0 conn(0x7fad780a4350 legacy=0x7fad781d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.152:6790/0 conn(0x7fad780a3860 legacy=0x7fad781cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 1 monclient: found mon.d 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7806bb00 con 0x7fad7806c1d0 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7cae9640 5 monclient: authenticate success, global_id 94155 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7cae9640 10 monclient: _renew_subs 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7cae9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad781abd00 con 0x7fad7806c1d0 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7cae9640 10 monclient: _renew_subs 2024-09-16T10:31:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7cae9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad781aa5a0 con 0x7fad7806c1d0 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad68003b30 con 0x7fad7806c1d0 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fad68003fe0 con 0x7fad7806c1d0 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7559a640 4 set_mon_vals no callback set 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 10 monclient: dump: 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.249 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7cae9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fad5c006580 con 0x7fad5c002820 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.229+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fad680041c0 con 0x7fad7806c1d0 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.230+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7fad68045fa0 con 0x7fad7806c1d0 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.230+0000 7fad76d9d640 10 monclient: get_auth_request con 0x7fad58039aa0 auth_method 0 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6dd9b640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.152:6790/0 conn(0x7fad5c002820 legacy=0x7fad5c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923922125 0 0) 0x7fad5c006580 con 0x7fad5c002820 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 10 monclient(hunting): my global_id is 4809 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad7809f270 con 0x7fad5c002820 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2965848742 0 0) 0x7fad7809f270 con 0x7fad5c002820 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad5c006580 con 0x7fad5c002820 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fad60001560 con 0x7fad5c002820 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.232+0000 7fad6f7fe640 10 monclient: dump: 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.251 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.233+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1917130040 0 0) 0x7fad5c006580 con 0x7fad5c002820 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.233+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.152:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad5c00a0d0 con 0x7fad5c002820 2024-09-16T10:31:50.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.233+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fad600019f0 con 0x7fad5c002820 2024-09-16T10:31:50.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.233+0000 7fad6f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.233+0000 7fad6f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:50.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.233+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.152:6790/0 conn(0x7fad5c002820 legacy=0x7fad5c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.233+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.152:6790/0 conn(0x7fad5c002820 legacy=0x7fad5c004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:50.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.239+0000 7fad7cae9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:31:50.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.239+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fad5c009d90 con 0x7fad5c0090b0 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6dd9b640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.152:6790/0 conn(0x7fad5c0090b0 legacy=0x7fad5c009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847315845 0 0) 0x7fad5c009d90 con 0x7fad5c0090b0 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 10 monclient(hunting): my global_id is 4818 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5c006580 con 0x7fad5c0090b0 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 204478442 0 0) 0x7fad5c006580 con 0x7fad5c0090b0 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad5c009d90 con 0x7fad5c0090b0 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fad60005530 con 0x7fad5c0090b0 2024-09-16T10:31:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 10 monclient: dump: 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.256 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1035660467 0 0) 0x7fad5c009d90 con 0x7fad5c0090b0 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 --> v1:172.21.15.152:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fad5c00c280 con 0x7fad5c0090b0 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 <== mon.3 v1:172.21.15.152:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7618 (unknown 0 0 3397637548) 0x7fad600078b0 con 0x7fad5c0090b0 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:31:50.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:50.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.152:6790/0 conn(0x7fad5c0090b0 legacy=0x7fad5c009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.242+0000 7fad6f7fe640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.152:6790/0 conn(0x7fad5c0090b0 legacy=0x7fad5c009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:50.258 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:50.258 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:50.258 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:50.258 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:31:50.258 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 18, 2024-09-16T10:31:50.258 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:31:50.258 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 1, 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:50.259 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.260 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.261 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:31:50.262 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.263 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.267 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:50.270 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.271 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:31:50.272 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.274 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.275 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.276 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.277 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.278 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.280 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:31:50.280 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:50.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.152:6800/2686360781 conn(0x7fad58039aa0 legacy=0x7fad5803bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad7cae9640 10 monclient: shutdown 2024-09-16T10:31:50.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad7cae9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 >> v1:172.21.15.94:6792/0 conn(0x7fad7806c1d0 legacy=0x7fad781be620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad76d9d640 1 -- 172.21.15.94:0/2389373598 reap_dead start 2024-09-16T10:31:50.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad76d9d640 1 -- 172.21.15.94:0/2389373598 reap_dead start 2024-09-16T10:31:50.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 shutdown_connections 2024-09-16T10:31:50.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 >> 172.21.15.94:0/2389373598 conn(0x7fad780a5960 msgr2=0x7fad780a7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 shutdown_connections 2024-09-16T10:31:50.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.243+0000 7fad7cae9640 1 -- 172.21.15.94:0/2389373598 wait complete. 2024-09-16T10:31:50.282 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:31:50.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-16T10:31:50.282 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-16T10:31:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.378+0000 7ff3ffd1f640 10 monclient: get_monmap_and_config 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.378+0000 7ff3ffd1f640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.378+0000 7ff3ffd1f640 10 monclient: monmap: 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:50.379216+0000 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:50.379216+0000 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.380 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.381 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.381 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.381 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.381 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.381 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.381 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.381 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.378+0000 7ff3ffd1f640 1 Processor -- start 2024-09-16T10:31:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 1 -- start start 2024-09-16T10:31:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 10 monclient: init 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 10 monclient: _add_conns ranks=[3,4,2,7,0,6,5,1,8] 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 10 monclient(hunting): picked mon.noname-g con 0x7ff3f81368e0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f8136cc0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f81370a0 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f8061ae0 con 0x7ff3f81370a0 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f80a03e0 con 0x7ff3f81368e0 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f806c8d0 con 0x7ff3f8136cc0 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3ffd1f640 10 monclient(hunting): authenticate will time out at 877.719716s 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3fe2bc640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7ff3f81370a0 0x7ff3f813ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41090/0 (socket says 172.21.15.94:41090) 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3fe2bc640 1 -- 172.21.15.94:0/3995752914 learned_addr learned my addr 172.21.15.94:0/3995752914 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.379+0000 7ff3fd2ba640 1 --1- 172.21.15.94:0/3995752914 >> v1:172.21.15.94:6791/0 conn(0x7ff3f8136cc0 0x7ff3f813b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:34000/0 (socket says 172.21.15.94:34000) 2024-09-16T10:31:50.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.381+0000 7ff3fd2ba640 1 -- 172.21.15.94:0/3995752914 >> v1:172.21.15.94:6791/0 conn(0x7ff3f8136cc0 legacy=0x7ff3f813b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fe2bc640 1 -- 172.21.15.94:0/3995752914 >> v1:172.21.15.94:6790/0 conn(0x7ff3f81370a0 legacy=0x7ff3f813ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409992524 0 0) 0x7ff3f806c8d0 con 0x7ff3f8136cc0 2024-09-16T10:31:50.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient(hunting): my global_id is 4621 2024-09-16T10:31:50.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e4003520 con 0x7ff3f8136cc0 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fdabb640 1 -- 172.21.15.94:0/3995752914 >> v1:172.21.15.152:6790/0 conn(0x7ff3f81368e0 legacy=0x7ff3f80a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834179895 0 0) 0x7ff3f8061ae0 con 0x7ff3f81370a0 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient(hunting): my global_id is 4574 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f806c8d0 con 0x7ff3f81370a0 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 463115825 0 0) 0x7ff3e4003520 con 0x7ff3f8136cc0 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 >> v1:172.21.15.152:6790/0 conn(0x7ff3f81368e0 legacy=0x7ff3f80a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 >> v1:172.21.15.94:6790/0 conn(0x7ff3f81370a0 legacy=0x7ff3f813ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 monclient: found mon.noname-c 2024-09-16T10:31:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f80a0680 con 0x7ff3f8136cc0 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff3f4001a60 con 0x7ff3f8136cc0 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3ffd1f640 5 monclient: authenticate success, global_id 4621 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3ffd1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient: dump: 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.390 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3ffd1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fd2ba640 1 --1- 172.21.15.94:0/3995752914 >> v1:172.21.15.94:6791/0 conn(0x7ff3f8136cc0 0x7ff3f813b7d0 :-1 s=OPENED pgs=161 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7ff3f4001d70 config(0 keys) v1 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f4001d70 con 0x7ff3f8136cc0 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3ffd1f640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3ffd1f640 4 set_mon_vals no callback set 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3ffd1f640 10 monclient: shutdown 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3ffd1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/3995752914 >> v1:172.21.15.94:6791/0 conn(0x7ff3f8136cc0 legacy=0x7ff3f813b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 1 -- 172.21.15.94:0/3995752914 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff3f4004b20 con 0x7ff3f8136cc0 2024-09-16T10:31:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.382+0000 7ff3fcab9640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.383+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/3995752914 shutdown_connections 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.383+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/3995752914 >> 172.21.15.94:0/3995752914 conn(0x7ff3f80a58d0 msgr2=0x7ff3f80a5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.383+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/3995752914 shutdown_connections 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.383+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/3995752914 wait complete. 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.383+0000 7ff3ffd1f640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.383+0000 7ff3ffd1f640 10 monclient: monmap: 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.383+0000 7ff3ffd1f640 1 Processor -- start 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 1 -- start start 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 10 monclient: init 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 10 monclient: _add_conns ranks=[4,1,3,7,0,8,2,6,5] 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f81368e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 10 monclient(hunting): picked mon.noname-f con 0x7ff3f8136cc0 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 10 monclient(hunting): picked mon.noname-g con 0x7ff3f81370a0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f8123210 con 0x7ff3f8136cc0 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f8121d60 con 0x7ff3f81370a0 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f81211b0 con 0x7ff3f81368e0 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3ffd1f640 10 monclient(hunting): authenticate will time out at 877.724476s 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3fdabb640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7ff3f81368e0 0x7ff3f81caae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:34014/0 (socket says 172.21.15.94:34014) 2024-09-16T10:31:50.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.384+0000 7ff3fdabb640 1 -- 172.21.15.94:0/4235463981 learned_addr learned my addr 172.21.15.94:0/4235463981 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3fdabb640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.94:6791/0 conn(0x7ff3f81368e0 legacy=0x7ff3f81caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329942227 0 0) 0x7ff3f81211b0 con 0x7ff3f81368e0 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3ea7fc640 10 monclient(hunting): my global_id is 4630 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc003520 con 0x7ff3f81368e0 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3fdabb640 1 --1- 172.21.15.94:0/4235463981 >> v1:172.21.15.94:6791/0 conn(0x7ff3f81368e0 0x7ff3f81caae0 :-1 s=OPENED pgs=162 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7ff3dc003520 auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3474660763 0 0) 0x7ff3dc003520 con 0x7ff3f81368e0 2024-09-16T10:31:50.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f81211b0 con 0x7ff3f81368e0 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3fd2ba640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6789/0 conn(0x7ff3f8136cc0 legacy=0x7ff3f8135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.386+0000 7ff3fe2bc640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6790/0 conn(0x7ff3f81370a0 legacy=0x7ff3f81d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff3ec001d30 con 0x7ff3f81368e0 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): dump: 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15113501 0 0) 0x7ff3f8121d60 con 0x7ff3f81370a0 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): my global_id is 4836 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc003520 con 0x7ff3f81370a0 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832527723 0 0) 0x7ff3f8123210 con 0x7ff3f8136cc0 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): my global_id is 4564 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f8121d60 con 0x7ff3f8136cc0 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2135717343 0 0) 0x7ff3dc003520 con 0x7ff3f81370a0 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f8123210 con 0x7ff3f81370a0 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff3f0001f60 con 0x7ff3f81370a0 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): dump: 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.404 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4094357286 0 0) 0x7ff3f8121d60 con 0x7ff3f8136cc0 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3dc003520 con 0x7ff3f8136cc0 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff3f4004b20 con 0x7ff3f8136cc0 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:31:50.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient(hunting): dump: 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.406 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2495189248 0 0) 0x7ff3f8123210 con 0x7ff3f81370a0 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.94:6791/0 conn(0x7ff3f81368e0 legacy=0x7ff3f81caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6789/0 conn(0x7ff3f8136cc0 legacy=0x7ff3f8135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 monclient: found mon.g 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f80a2390 con 0x7ff3f81370a0 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ffd1f640 5 monclient: authenticate success, global_id 4836 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2873637288 0 0) 0x7ff3f81211b0 con 0x7ff3f81368e0 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ffd1f640 10 monclient: _renew_subs 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ffd1f640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f80a1ed0 con 0x7ff3f81370a0 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ffd1f640 10 monclient: _renew_subs 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ffd1f640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:31:50.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f8121ab0 con 0x7ff3f81370a0 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f0004e10 con 0x7ff3f81370a0 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff3f00052c0 con 0x7ff3f81370a0 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ebfff640 4 set_mon_vals no callback set 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.387+0000 7ff3ea7fc640 10 monclient: dump: 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.409 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.388+0000 7ff3ffd1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.388+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3e0006580 con 0x7ff3e0002820 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.388+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7ff3f0005420 con 0x7ff3f81370a0 2024-09-16T10:31:50.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.388+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7ff3f00460d0 con 0x7ff3f81370a0 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.389+0000 7ff3fd2ba640 10 monclient: get_auth_request con 0x7ff3dc039bf0 auth_method 0 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.390+0000 7ff3fdabb640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6791/0 conn(0x7ff3e0002820 legacy=0x7ff3e0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863437834 0 0) 0x7ff3e0006580 con 0x7ff3e0002820 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 10 monclient(hunting): my global_id is 4640 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f81211b0 con 0x7ff3e0002820 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 530995139 0 0) 0x7ff3f81211b0 con 0x7ff3e0002820 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3e0006580 con 0x7ff3e0002820 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff3ec004260 con 0x7ff3e0002820 2024-09-16T10:31:50.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 10 monclient: dump: 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.412 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4079858148 0 0) 0x7ff3e0006580 con 0x7ff3e0002820 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3e000a0d0 con 0x7ff3e0002820 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff3ec0058f0 con 0x7ff3e0002820 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6791/0 conn(0x7ff3e0002820 legacy=0x7ff3e0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.391+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6791/0 conn(0x7ff3e0002820 legacy=0x7ff3e0004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:50.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.397+0000 7ff3ffd1f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:31:50.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.397+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3e0009d70 con 0x7ff3e0009090 2024-09-16T10:31:50.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3fdabb640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6791/0 conn(0x7ff3e0009090 legacy=0x7ff3e0009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195998689 0 0) 0x7ff3e0009d70 con 0x7ff3e0009090 2024-09-16T10:31:50.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 10 monclient(hunting): my global_id is 4649 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e0006580 con 0x7ff3e0009090 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 222635789 0 0) 0x7ff3e0006580 con 0x7ff3e0009090 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3e0009d70 con 0x7ff3e0009090 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff3ec006020 con 0x7ff3e0009090 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.400+0000 7ff3ea7fc640 10 monclient: dump: 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.416 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 480897216 0 0) 0x7ff3e0009d70 con 0x7ff3e0009090 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 --> v1:172.21.15.152:6791/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff3e000c280 con 0x7ff3e0009090 2024-09-16T10:31:50.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 <== mon.5 v1:172.21.15.152:6791/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7618 (unknown 0 0 3289911613) 0x7ff3ec0082c0 con 0x7ff3e0009090 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6791/0 conn(0x7ff3e0009090 legacy=0x7ff3e0009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ea7fc640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6791/0 conn(0x7ff3e0009090 legacy=0x7ff3e0009520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 18, 2024-09-16T10:31:50.418 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 1, 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:50.419 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.420 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.421 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:31:50.422 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.423 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.424 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.425 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.427 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:50.428 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:31:50.429 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.434 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:31:50.435 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:50.436 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:31:50.437 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.438 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": [ 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:31:50.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.440 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.440 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:31:50.440 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:50.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6800/2686360781 conn(0x7ff3dc039bf0 legacy=0x7ff3dc03c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ffd1f640 10 monclient: shutdown 2024-09-16T10:31:50.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ffd1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 >> v1:172.21.15.152:6790/0 conn(0x7ff3f81370a0 legacy=0x7ff3f81d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3fe2bc640 1 -- 172.21.15.94:0/4235463981 reap_dead start 2024-09-16T10:31:50.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3fe2bc640 1 -- 172.21.15.94:0/4235463981 reap_dead start 2024-09-16T10:31:50.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 shutdown_connections 2024-09-16T10:31:50.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 >> 172.21.15.94:0/4235463981 conn(0x7ff3f80a58d0 msgr2=0x7ff3f81be2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 shutdown_connections 2024-09-16T10:31:50.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.401+0000 7ff3ffd1f640 1 -- 172.21.15.94:0/4235463981 wait complete. 2024-09-16T10:31:50.442 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:31:50.442 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-16T10:31:50.442 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-16T10:31:50.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.538+0000 7f8f7ecc9640 10 monclient: get_monmap_and_config 2024-09-16T10:31:50.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.538+0000 7f8f7ecc9640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.538+0000 7f8f7ecc9640 10 monclient: monmap: 2024-09-16T10:31:50.540 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.540 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.540 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:50.539312+0000 2024-09-16T10:31:50.540 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:50.539312+0000 2024-09-16T10:31:50.540 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.540 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.541 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 1 Processor -- start 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 1 -- start start 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 10 monclient: init 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 10 monclient: _add_conns ranks=[0,8,6,1,3,4,7,2,5] 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 10 monclient(hunting): picked mon.noname-a con 0x7f8f780a3860 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 10 monclient(hunting): picked mon.noname-e con 0x7f8f7806c1d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 10 monclient(hunting): picked mon.noname-d con 0x7f8f780a4350 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f78061ae0 con 0x7f8f780a3860 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780a0600 con 0x7f8f780a4350 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780a0b30 con 0x7f8f7806c1d0 2024-09-16T10:31:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ecc9640 10 monclient(hunting): authenticate will time out at 877.879764s 2024-09-16T10:31:50.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f77fff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f8f7806c1d0 0x7f8f7813b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:44070/0 (socket says 172.21.15.94:44070) 2024-09-16T10:31:50.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f77fff640 1 -- 172.21.15.94:0/2311797257 learned_addr learned my addr 172.21.15.94:0/2311797257 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7d266640 1 --1- 172.21.15.94:0/2311797257 >> v1:172.21.15.94:6792/0 conn(0x7f8f780a4350 0x7f8f7813ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:49006/0 (socket says 172.21.15.94:49006) 2024-09-16T10:31:50.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.539+0000 7f8f7ca65640 1 --1- 172.21.15.94:0/2311797257 >> v1:172.21.15.94:6789/0 conn(0x7f8f780a3860 0x7f8f780a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53444/0 (socket says 172.21.15.94:53444) 2024-09-16T10:31:50.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f77fff640 1 -- 172.21.15.94:0/2311797257 >> v1:172.21.15.94:6793/0 conn(0x7f8f7806c1d0 legacy=0x7f8f7813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f7ca65640 1 -- 172.21.15.94:0/2311797257 >> v1:172.21.15.94:6789/0 conn(0x7f8f780a3860 legacy=0x7f8f780a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f7d266640 1 -- 172.21.15.94:0/2311797257 >> v1:172.21.15.94:6792/0 conn(0x7f8f780a4350 legacy=0x7f8f7813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165701208 0 0) 0x7f8f78061ae0 con 0x7f8f780a3860 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): my global_id is 4563 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f68003520 con 0x7f8f780a3860 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903098244 0 0) 0x7f8f780a0600 con 0x7f8f780a4350 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): my global_id is 94164 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f78061ae0 con 0x7f8f780a4350 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238155931 0 0) 0x7f8f780a0b30 con 0x7f8f7806c1d0 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient(hunting): my global_id is 4616 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f780a0600 con 0x7f8f7806c1d0 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1482782977 0 0) 0x7f8f68003520 con 0x7f8f780a3860 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 >> v1:172.21.15.94:6793/0 conn(0x7f8f7806c1d0 legacy=0x7f8f7813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 >> v1:172.21.15.94:6792/0 conn(0x7f8f780a4350 legacy=0x7f8f7813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 monclient: found mon.noname-a 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f780a0e10 con 0x7f8f780a3860 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8f6c001a50 con 0x7f8f780a3860 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: dump: 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.550 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.551 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f7ecc9640 5 monclient: authenticate success, global_id 4563 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f7ecc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f6c001d60 con 0x7f8f780a3860 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 1 -- 172.21.15.94:0/2311797257 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8f6c004b20 con 0x7f8f780a3860 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f777fe640 10 monclient: dump: 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f7ecc9640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:50.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.542+0000 7f8f7ecc9640 4 set_mon_vals no callback set 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.543+0000 7f8f7ecc9640 10 monclient: shutdown 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.543+0000 7f8f7ecc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.543+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2311797257 >> v1:172.21.15.94:6789/0 conn(0x7f8f780a3860 legacy=0x7f8f780a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.543+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2311797257 shutdown_connections 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.543+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2311797257 >> 172.21.15.94:0/2311797257 conn(0x7f8f780a5960 msgr2=0x7f8f780a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.543+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2311797257 shutdown_connections 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.543+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2311797257 wait complete. 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.543+0000 7f8f7ecc9640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.543+0000 7f8f7ecc9640 10 monclient: monmap: 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:50.554 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 1 Processor -- start 2024-09-16T10:31:50.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 1 -- start start 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 10 monclient: init 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 10 monclient: _add_conns ranks=[1,2,4,0,6,5,3,7,8] 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 10 monclient(hunting): picked mon.noname-f con 0x7f8f7806c1d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 10 monclient(hunting): picked mon.noname-b con 0x7f8f780a3860 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 10 monclient(hunting): picked mon.noname-c con 0x7f8f780a4350 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7809eef0 con 0x7f8f7806c1d0 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f781c4e80 con 0x7f8f780a3860 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7806b7c0 con 0x7f8f780a4350 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7ecc9640 10 monclient(hunting): authenticate will time out at 877.884786s 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f77fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f8f780a3860 0x7f8f781cfaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41094/0 (socket says 172.21.15.94:41094) 2024-09-16T10:31:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f7d266640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f8f780a4350 0x7f8f781d31d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:34026/0 (socket says 172.21.15.94:34026) 2024-09-16T10:31:50.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.544+0000 7f8f77fff640 1 -- 172.21.15.94:0/2009704922 learned_addr learned my addr 172.21.15.94:0/2009704922 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f77fff640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.94:6790/0 conn(0x7f8f780a3860 legacy=0x7f8f781cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f7d266640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.94:6791/0 conn(0x7f8f780a4350 legacy=0x7f8f781d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215360374 0 0) 0x7f8f7806b7c0 con 0x7f8f780a4350 2024-09-16T10:31:50.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): my global_id is 4639 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f5c003520 con 0x7f8f780a4350 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291319342 0 0) 0x7f8f781c4e80 con 0x7f8f780a3860 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): my global_id is 4583 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f7806b7c0 con 0x7f8f780a3860 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4193485028 0 0) 0x7f8f7806b7c0 con 0x7f8f780a3860 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f781c4e80 con 0x7f8f780a3860 2024-09-16T10:31:50.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f7ca65640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6789/0 conn(0x7f8f7806c1d0 legacy=0x7f8f781c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8f64001cb0 con 0x7f8f780a3860 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): dump: 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.563 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3461483672 0 0) 0x7f8f5c003520 con 0x7f8f780a4350 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f7806b7c0 con 0x7f8f780a4350 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8f70001ff0 con 0x7f8f780a4350 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient(hunting): dump: 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.565 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3055076960 0 0) 0x7f8f781c4e80 con 0x7f8f780a3860 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.94:6791/0 conn(0x7f8f780a4350 legacy=0x7f8f781d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6789/0 conn(0x7f8f7806c1d0 legacy=0x7f8f781c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 monclient: found mon.b 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f781a95a0 con 0x7f8f780a3860 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f7ecc9640 5 monclient: authenticate success, global_id 4583 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f7ecc9640 10 monclient: _renew_subs 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f7ecc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f781ac0d0 con 0x7f8f780a3860 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f7ecc9640 10 monclient: _renew_subs 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f7ecc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f781aab80 con 0x7f8f780a3860 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f64004dd0 con 0x7f8f780a3860 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8f64005280 con 0x7f8f780a3860 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f76ffd640 4 set_mon_vals no callback set 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.547+0000 7f8f757fa640 10 monclient: dump: 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.568 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.548+0000 7f8f7ecc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.548+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8f58006580 con 0x7f8f58002820 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.548+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f8f640102e0 con 0x7f8f780a3860 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.548+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f8f64045fa0 con 0x7f8f780a3860 2024-09-16T10:31:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.548+0000 7f8f7d266640 10 monclient: get_auth_request con 0x7f8f5c039710 auth_method 0 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.550+0000 7f8f7ca65640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6792/0 conn(0x7f8f58002820 legacy=0x7f8f58004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340749434 0 0) 0x7f8f58006580 con 0x7f8f58002820 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 10 monclient(hunting): my global_id is 4642 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f781c4e80 con 0x7f8f58002820 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f7d266640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8f5c039710 legacy=0x7f8f5c03bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f7d266640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8f5c039710 legacy=0x7f8f5c03bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f7d266640 1 --1- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8f5c039710 0x7f8f5c03bbd0 :-1 s=OPENED pgs=98 cs=1 l=1).handle_message read tag failed 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f7d266640 1 --1- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8f5c039710 0x7f8f5c03bbd0 :-1 s=OPENED pgs=98 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:31:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8f5c039710 legacy=0x7f8f5c03bbd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3727255792 0 0) 0x7f8f781c4e80 con 0x7f8f58002820 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f58006580 con 0x7f8f58002820 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8f6c004b20 con 0x7f8f58002820 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 10 monclient: dump: 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2509304531 0 0) 0x7f8f58006580 con 0x7f8f58002820 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.152:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f5800a0d0 con 0x7f8f58002820 2024-09-16T10:31:50.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8f6c001880 con 0x7f8f58002820 2024-09-16T10:31:50.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:50.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6792/0 conn(0x7f8f58002820 legacy=0x7f8f58004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.551+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6792/0 conn(0x7f8f58002820 legacy=0x7f8f58004cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:50.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.557+0000 7f8f7ecc9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:31:50.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.557+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8f58009d60 con 0x7f8f580090b0 2024-09-16T10:31:50.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.560+0000 7f8f7ca65640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6792/0 conn(0x7f8f580090b0 legacy=0x7f8f58009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.560+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350459721 0 0) 0x7f8f58009d60 con 0x7f8f580090b0 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.560+0000 7f8f757fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.560+0000 7f8f757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.560+0000 7f8f757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.560+0000 7f8f757fa640 10 monclient(hunting): my global_id is 4651 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.560+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f58006580 con 0x7f8f580090b0 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1386329333 0 0) 0x7f8f58006580 con 0x7f8f580090b0 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f58009d60 con 0x7f8f580090b0 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8f6c001a70 con 0x7f8f580090b0 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 10 monclient: dump: 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.576 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 946891816 0 0) 0x7f8f58009d60 con 0x7f8f580090b0 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 --> v1:172.21.15.152:6792/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8f5800c280 con 0x7f8f580090b0 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 <== mon.7 v1:172.21.15.152:6792/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7452 (unknown 0 0 9709738) 0x7f8f6c0076f0 con 0x7f8f580090b0 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:50.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6792/0 conn(0x7f8f580090b0 legacy=0x7f8f58009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f757fa640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.152:6792/0 conn(0x7f8f580090b0 legacy=0x7f8f58009540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f7d266640 1 -- 172.21.15.94:0/2009704922 reap_dead start 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "peon", 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stdout: "election_epoch": 18, 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum": [ 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stdout: 0, 2024-09-16T10:31:50.578 INFO:teuthology.orchestra.run.smithi094.stdout: 1, 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: 2, 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: 3, 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: 4, 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: 5, 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: 6, 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: 7, 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: 8 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_age": 1, 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: "required_con": "2449958758054445060", 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: "required_mon": [ 2024-09-16T10:31:50.579 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "quorum_mon": [ 2024-09-16T10:31:50.580 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.581 INFO:teuthology.orchestra.run.smithi094.stdout: "outside_quorum": [], 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "extra_probe_peers": [], 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_provider": [], 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "monmap": { 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "fsid": "f51dbe5c-da09-4bb3-8cd3-867d68869245", 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "modified": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "created": "2024-09-16T10:28:46.257482Z", 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release": 18, 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "min_mon_release_name": "reef", 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "election_strategy": 1, 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "disallowed_leaders: ": "", 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false, 2024-09-16T10:31:50.582 INFO:teuthology.orchestra.run.smithi094.stdout: "tiebreaker_mon": "", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "removed_ranks: ": "", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "features": { 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "persistent": [ 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "kraken", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "luminous", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "mimic", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap-prune", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "nautilus", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "elector-pinging", 2024-09-16T10:31:50.583 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy", 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: "reef" 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: "optional": [] 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: "mons": [ 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 0, 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "a", 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.584 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789", 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6789/0", 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6789/0", 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 1, 2024-09-16T10:31:50.585 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "f", 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789", 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6789/0", 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6789/0", 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 2, 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "b", 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790", 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6790/0", 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6790/0", 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 3, 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "g", 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790", 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6790/0", 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6790/0", 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 4, 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "c", 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791", 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.590 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6791/0", 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6791/0", 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 5, 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "h", 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.591 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791", 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6791/0", 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6791/0", 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.592 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 6, 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "d", 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792", 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6792/0", 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6792/0", 2024-09-16T10:31:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 7, 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "i", 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792", 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.152:6792/0", 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.152:6792/0", 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: "rank": 8, 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "e", 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addrs": { 2024-09-16T10:31:50.595 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793", 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 0 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6793/0", 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: "public_addr": "172.21.15.94:6793/0", 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: "priority": 0, 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: "weight": 0, 2024-09-16T10:31:50.596 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_location": "{}" 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: "feature_map": { 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": [ 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 1 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-09-16T10:31:50.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client": [ 2024-09-16T10:31:50.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-16T10:31:50.598 INFO:teuthology.orchestra.run.smithi094.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:31:50.598 INFO:teuthology.orchestra.run.smithi094.stdout: "release": "reef", 2024-09-16T10:31:50.598 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 2 2024-09-16T10:31:50.598 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-16T10:31:50.598 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-16T10:31:50.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-16T10:31:50.598 INFO:teuthology.orchestra.run.smithi094.stdout: "stretch_mode": false 2024-09-16T10:31:50.598 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:50.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f7ecc9640 10 monclient: shutdown 2024-09-16T10:31:50.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f7ecc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.561+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2009704922 >> v1:172.21.15.94:6790/0 conn(0x7f8f780a3860 legacy=0x7f8f781cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.562+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2009704922 shutdown_connections 2024-09-16T10:31:50.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.562+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2009704922 >> 172.21.15.94:0/2009704922 conn(0x7f8f780a5960 msgr2=0x7f8f7809ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.562+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2009704922 shutdown_connections 2024-09-16T10:31:50.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.562+0000 7f8f7ecc9640 1 -- 172.21.15.94:0/2009704922 wait complete. 2024-09-16T10:31:50.600 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-16T10:31:50.601 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T10:31:50.601 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T10:31:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.695+0000 7fbd338c5640 10 monclient: get_monmap_and_config 2024-09-16T10:31:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.695+0000 7fbd338c5640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.695+0000 7fbd338c5640 10 monclient: monmap: 2024-09-16T10:31:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:50.696382+0000 2024-09-16T10:31:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:50.696382+0000 2024-09-16T10:31:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.698 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 1 Processor -- start 2024-09-16T10:31:50.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 1 -- start start 2024-09-16T10:31:50.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 10 monclient: init 2024-09-16T10:31:50.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 10 monclient: _add_conns ranks=[8,6,0,4,2,1,7,3,5] 2024-09-16T10:31:50.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 10 monclient(hunting): picked mon.noname-e con 0x7fbd2c0a3840 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:50.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 10 monclient(hunting): picked mon.noname-d con 0x7fbd2c0a3c20 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:50.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 10 monclient(hunting): picked mon.noname-a con 0x7fbd2c13ca60 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:50.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c061ae0 con 0x7fbd2c13ca60 2024-09-16T10:31:50.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c0a52c0 con 0x7fbd2c0a3c20 2024-09-16T10:31:50.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c06c470 con 0x7fbd2c0a3840 2024-09-16T10:31:50.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.696+0000 7fbd338c5640 10 monclient(hunting): authenticate will time out at 878.036925s 2024-09-16T10:31:50.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.697+0000 7fbd30e60640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fbd2c0a3c20 0x7fbd2c0a4000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:49022/0 (socket says 172.21.15.94:49022) 2024-09-16T10:31:50.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.697+0000 7fbd31661640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7fbd2c0a3840 0x7fbd2c0a1060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:44080/0 (socket says 172.21.15.94:44080) 2024-09-16T10:31:50.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.697+0000 7fbd31e62640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbd2c13ca60 0x7fbd2c13ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53460/0 (socket says 172.21.15.94:53460) 2024-09-16T10:31:50.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.697+0000 7fbd30e60640 1 -- 172.21.15.94:0/3725823351 learned_addr learned my addr 172.21.15.94:0/3725823351 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd31661640 1 -- 172.21.15.94:0/3725823351 >> v1:172.21.15.94:6793/0 conn(0x7fbd2c0a3840 legacy=0x7fbd2c0a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd31e62640 1 -- 172.21.15.94:0/3725823351 >> v1:172.21.15.94:6789/0 conn(0x7fbd2c13ca60 legacy=0x7fbd2c13ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd30e60640 1 -- 172.21.15.94:0/3725823351 >> v1:172.21.15.94:6792/0 conn(0x7fbd2c0a3c20 legacy=0x7fbd2c0a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176126468 0 0) 0x7fbd2c0a52c0 con 0x7fbd2c0a3c20 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): my global_id is 94173 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd18003520 con 0x7fbd2c0a3c20 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195983357 0 0) 0x7fbd2c061ae0 con 0x7fbd2c13ca60 2024-09-16T10:31:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): my global_id is 4572 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd2c0a52c0 con 0x7fbd2c13ca60 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363060628 0 0) 0x7fbd2c06c470 con 0x7fbd2c0a3840 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient(hunting): my global_id is 4625 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd2c061ae0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4205991757 0 0) 0x7fbd2c061ae0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 >> v1:172.21.15.94:6792/0 conn(0x7fbd2c0a3c20 legacy=0x7fbd2c0a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 >> v1:172.21.15.94:6789/0 conn(0x7fbd2c13ca60 legacy=0x7fbd2c13ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 monclient: found mon.noname-e 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c06c6b0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd338c5640 5 monclient: authenticate success, global_id 4625 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd28001a00 con 0x7fbd2c0a3840 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd338c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: dump: 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.710 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd338c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd28001c50 con 0x7fbd2c0a3840 2024-09-16T10:31:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 1 -- 172.21.15.94:0/3725823351 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd28004b20 con 0x7fbd2c0a3840 2024-09-16T10:31:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd338c5640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd338c5640 4 set_mon_vals no callback set 2024-09-16T10:31:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.699+0000 7fbd23fff640 10 monclient: dump: 2024-09-16T10:31:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.712 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.700+0000 7fbd338c5640 10 monclient: shutdown 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.700+0000 7fbd338c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.700+0000 7fbd338c5640 1 -- 172.21.15.94:0/3725823351 >> v1:172.21.15.94:6793/0 conn(0x7fbd2c0a3840 legacy=0x7fbd2c0a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.700+0000 7fbd338c5640 1 -- 172.21.15.94:0/3725823351 shutdown_connections 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.700+0000 7fbd338c5640 1 -- 172.21.15.94:0/3725823351 >> 172.21.15.94:0/3725823351 conn(0x7fbd2c0a9530 msgr2=0x7fbd2c0a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.700+0000 7fbd338c5640 1 -- 172.21.15.94:0/3725823351 shutdown_connections 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.700+0000 7fbd338c5640 1 -- 172.21.15.94:0/3725823351 wait complete. 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.700+0000 7fbd338c5640 10 monclient: build_initial_monmap 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.700+0000 7fbd338c5640 10 monclient: monmap: 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:50.714 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 1 Processor -- start 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 1 -- start start 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 10 monclient: init 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 10 monclient: _add_conns ranks=[4,5,3,0,1,7,6,2,8] 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 10 monclient(hunting): picked mon.noname-c con 0x7fbd2c0a3840 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 10 monclient(hunting): picked mon.noname-h con 0x7fbd2c0a3c20 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 10 monclient(hunting): picked mon.noname-g con 0x7fbd2c13ca60 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c124ec0 con 0x7fbd2c13ca60 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c0a72f0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c0a6da0 con 0x7fbd2c0a3c20 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:50.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd338c5640 10 monclient(hunting): authenticate will time out at 878.041817s 2024-09-16T10:31:50.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd31661640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fbd2c0a3840 0x7fbd2c1cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:34036/0 (socket says 172.21.15.94:34036) 2024-09-16T10:31:50.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.701+0000 7fbd31661640 1 -- 172.21.15.94:0/1986629370 learned_addr learned my addr 172.21.15.94:0/1986629370 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:50.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.703+0000 7fbd31661640 1 -- 172.21.15.94:0/1986629370 >> v1:172.21.15.94:6791/0 conn(0x7fbd2c0a3840 legacy=0x7fbd2c1cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.703+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 DispatchQueue::entry inject delay of 0.002000 2024-09-16T10:31:50.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.704+0000 7fbd30e60640 1 -- 172.21.15.94:0/1986629370 >> v1:172.21.15.152:6791/0 conn(0x7fbd2c0a3c20 legacy=0x7fbd2c1cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.704+0000 7fbd31e62640 1 -- 172.21.15.94:0/1986629370 >> v1:172.21.15.152:6790/0 conn(0x7fbd2c13ca60 legacy=0x7fbd2c1d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:50.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844198324 0 0) 0x7fbd2c0a72f0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): my global_id is 4648 2024-09-16T10:31:50.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd10003520 con 0x7fbd2c0a3840 2024-09-16T10:31:50.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873117280 0 0) 0x7fbd2c0a6da0 con 0x7fbd2c0a3c20 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): my global_id is 4658 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd2c0a72f0 con 0x7fbd2c0a3c20 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351614663 0 0) 0x7fbd2c124ec0 con 0x7fbd2c13ca60 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): my global_id is 4845 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd2c0a6da0 con 0x7fbd2c13ca60 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2485843483 0 0) 0x7fbd10003520 con 0x7fbd2c0a3840 2024-09-16T10:31:50.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd2c124ec0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd28004b20 con 0x7fbd2c0a3840 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): dump: 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.721 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 812696567 0 0) 0x7fbd2c0a72f0 con 0x7fbd2c0a3c20 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd10003520 con 0x7fbd2c0a3c20 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd1c001cc0 con 0x7fbd2c0a3c20 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): dump: 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.723 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2199278152 0 0) 0x7fbd2c0a6da0 con 0x7fbd2c13ca60 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd2c0a72f0 con 0x7fbd2c13ca60 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd24001af0 con 0x7fbd2c13ca60 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient(hunting): dump: 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.724 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.725 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3749921171 0 0) 0x7fbd2c124ec0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 >> v1:172.21.15.152:6791/0 conn(0x7fbd2c0a3c20 legacy=0x7fbd2c1cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 >> v1:172.21.15.152:6790/0 conn(0x7fbd2c13ca60 legacy=0x7fbd2c1d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 monclient: found mon.c 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c13bd90 con 0x7fbd2c0a3840 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient: _finish_auth 0 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd338c5640 5 monclient: authenticate success, global_id 4648 2024-09-16T10:31:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd338c5640 10 monclient: _renew_subs 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd338c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2c1251a0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd280050d0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fbd28005580 con 0x7fbd2c0a3840 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd21ffb640 10 monclient: dump: 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd338c5640 10 monclient: _renew_subs 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd237fe640 4 set_mon_vals no callback set 2024-09-16T10:31:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd338c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.706+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2c122dd0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.707+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fbd280105e0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.707+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7fbd28046080 con 0x7fbd2c0a3840 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.707+0000 7fbd338c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.707+0000 7fbd338c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.707+0000 7fbd338c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.707+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2c04e730 con 0x7fbd2c0a3840 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.708+0000 7fbd30e60640 10 monclient: get_auth_request con 0x7fbd10039d00 auth_method 0 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.712+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fbd280155f0 con 0x7fbd2c0a3840 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.712+0000 7fbd21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.712+0000 7fbd21ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:50.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.939+0000 7fbd338c5640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T10:31:50.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.939+0000 7fbd338c5640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T10:31:50.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.939+0000 7fbd338c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:50.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.939+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fbd2c0a6650 con 0x7fbd2c0a3840 2024-09-16T10:31:50.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.941+0000 7fbd21ffb640 1 -- 172.21.15.94:0/1986629370 <== mon.4 v1:172.21.15.94:6791/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fbd28013a60 con 0x7fbd2c0a3840 2024-09-16T10:31:50.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.941+0000 7fbd21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T10:31:50.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.941+0000 7fbd21ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:50.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.943+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 >> v1:172.21.15.152:6800/2686360781 conn(0x7fbd10039d00 legacy=0x7fbd1003c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.943+0000 7fbd338c5640 10 monclient: shutdown 2024-09-16T10:31:50.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.943+0000 7fbd338c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:50.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.943+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 >> v1:172.21.15.94:6791/0 conn(0x7fbd2c0a3840 legacy=0x7fbd2c1cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:50.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.943+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 shutdown_connections 2024-09-16T10:31:50.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.943+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 >> 172.21.15.94:0/1986629370 conn(0x7fbd2c0a9530 msgr2=0x7fbd2c0a8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:50.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.943+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 shutdown_connections 2024-09-16T10:31:50.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:50.943+0000 7fbd338c5640 1 -- 172.21.15.94:0/1986629370 wait complete. 2024-09-16T10:31:50.954 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=262681156,config=3541143443,health=3977944326,kv=1143020496,logm=3317411489}) 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=11,kv=10,logm=62} crc {auth=270606757,config=3541143443,health=1706377864,kv=584870117,logm=4113934511}) 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=262681156,config=3541143443,health=3977944326,kv=1143020496,logm=3317411489}) 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=11,kv=9,logm=62} crc {auth=262681156,config=3541143443,health=3926953126,kv=3656410872,logm=3413432109}) 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=262681156,config=3541143443,health=3977944326,kv=1143020496,logm=3317411489}) 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=16,config=2,health=12,kv=11,logm=59} crc {auth=262681156,config=3541143443,health=3977944326,kv=2876928431,logm=1595846906}) 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.017 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=262681156,config=3541143443,health=3977944326,kv=1143020496,logm=3317411489}) 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=15,config=2,health=11,kv=11,logm=61} crc {auth=4261193401,config=3541143443,health=1603706908,kv=2876928431,logm=948420809}) 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=262681156,config=3541143443,health=3977944326,kv=1143020496,logm=3317411489}) 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=15,config=2,health=11,kv=11,logm=61} crc {auth=3200195441,config=3541143443,health=1603706908,kv=2876928431,logm=2340753473}) 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=262681156,config=3541143443,health=3977944326,kv=1143020496,logm=3317411489}) 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=14,config=1,health=9,kv=11,logm=65} crc {auth=1800695304,config=555582061,health=3647517310,kv=2876928431,logm=2596819860}) 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=262681156,config=3541143443,health=3977944326,kv=1143020496,logm=3317411489}) 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=1838329163,config=3541143443,health=3977944326,kv=2876928431,logm=4148902194}) 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=262681156,config=3541143443,health=3977944326,kv=1143020496,logm=3317411489}) 2024-09-16T10:31:51.018 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.014+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=16,config=2,health=11,kv=10,logm=61} crc {auth=262681156,config=3541143443,health=3926953126,kv=1625779263,logm=2173801809}) 2024-09-16T10:31:51.183 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.183 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699031729}) 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2235475823}) 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699031729}) 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3233044571}) 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699031729}) 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1557799282}) 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699031729}) 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=20741399}) 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699031729}) 2024-09-16T10:31:51.184 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2542472728}) 2024-09-16T10:31:51.185 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.185 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699031729}) 2024-09-16T10:31:51.185 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1161608047}) 2024-09-16T10:31:51.185 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.185 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699031729}) 2024-09-16T10:31:51.185 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3359233981}) 2024-09-16T10:31:51.185 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.185 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699031729}) 2024-09-16T10:31:51.185 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.180+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=624231453}) 2024-09-16T10:31:51.189 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.189 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441184439}) 2024-09-16T10:31:51.189 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190656941}) 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441184439}) 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469498628}) 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441184439}) 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1605265664}) 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441184439}) 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1643458055}) 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441184439}) 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4115350068}) 2024-09-16T10:31:51.190 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.191 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441184439}) 2024-09-16T10:31:51.191 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1676727834}) 2024-09-16T10:31:51.191 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.191 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441184439}) 2024-09-16T10:31:51.191 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=392045200}) 2024-09-16T10:31:51.191 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.191 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441184439}) 2024-09-16T10:31:51.191 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.186+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=634083209}) 2024-09-16T10:31:51.195 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421025554}) 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1892089457}) 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421025554}) 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987855829}) 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421025554}) 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1059682577}) 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421025554}) 2024-09-16T10:31:51.196 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1108507275}) 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421025554}) 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2998333111}) 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421025554}) 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=757319633}) 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.192+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421025554}) 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.193+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1994369021}) 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.193+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.193+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421025554}) 2024-09-16T10:31:51.197 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.193+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=389958739}) 2024-09-16T10:31:51.452 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.452 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177294056}) 2024-09-16T10:31:51.452 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4043592504}) 2024-09-16T10:31:51.452 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.452 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177294056}) 2024-09-16T10:31:51.452 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001769334}) 2024-09-16T10:31:51.452 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177294056}) 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3925598733}) 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177294056}) 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2643853744}) 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177294056}) 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=801199555}) 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177294056}) 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2681987439}) 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.453 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177294056}) 2024-09-16T10:31:51.454 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2883734665}) 2024-09-16T10:31:51.454 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.454 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177294056}) 2024-09-16T10:31:51.454 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.449+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3659745223}) 2024-09-16T10:31:51.915 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.915 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135283947}) 2024-09-16T10:31:51.915 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2384387202}) 2024-09-16T10:31:51.915 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.915 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135283947}) 2024-09-16T10:31:51.915 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536799973}) 2024-09-16T10:31:51.915 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.915 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135283947}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1270219146}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135283947}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3161877433}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135283947}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1896732147}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135283947}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4055389150}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135283947}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4227871450}) 2024-09-16T10:31:51.916 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:51.917 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135283947}) 2024-09-16T10:31:51.917 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:51.912+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4287518731}) 2024-09-16T10:31:51.955 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-16T10:31:51.955 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:31:52.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.053+0000 7f8270c44640 10 monclient: get_monmap_and_config 2024-09-16T10:31:52.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.053+0000 7f8270c44640 10 monclient: build_initial_monmap 2024-09-16T10:31:52.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.053+0000 7f8270c44640 10 monclient: monmap: 2024-09-16T10:31:52.055 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:52.055 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.055 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:52.054439+0000 2024-09-16T10:31:52.055 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:52.054439+0000 2024-09-16T10:31:52.055 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:52.056 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 1 Processor -- start 2024-09-16T10:31:52.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 1 -- start start 2024-09-16T10:31:52.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 10 monclient: init 2024-09-16T10:31:52.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:52.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 10 monclient: _add_conns ranks=[3,5,7,2,8,0,6,4,1] 2024-09-16T10:31:52.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 10 monclient(hunting): picked mon.noname-g con 0x7f826c0a38d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:52.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 10 monclient(hunting): picked mon.noname-h con 0x7f826c06c670 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:52.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 10 monclient(hunting): picked mon.noname-i con 0x7f826c0a43c0 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:52.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c061ae0 con 0x7f826c0a38d0 2024-09-16T10:31:52.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0a04d0 con 0x7f826c06c670 2024-09-16T10:31:52.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0a3250 con 0x7f826c0a43c0 2024-09-16T10:31:52.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:52.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.054+0000 7f8270c44640 10 monclient(hunting): authenticate will time out at 879.394926s 2024-09-16T10:31:52.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.055+0000 7f8269d9b640 1 --1- >> v1:172.21.15.152:6791/0 conn(0x7f826c06c670 0x7f826c13bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.152:6791/0 says I am v1:172.21.15.94:33424/0 (socket says 172.21.15.94:33424) 2024-09-16T10:31:52.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.055+0000 7f8269d9b640 1 -- 172.21.15.94:0/4165805967 learned_addr learned my addr 172.21.15.94:0/4165805967 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f8269d9b640 1 -- 172.21.15.94:0/4165805967 >> v1:172.21.15.152:6791/0 conn(0x7f826c06c670 legacy=0x7f826c13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826ad9d640 1 -- 172.21.15.94:0/4165805967 >> v1:172.21.15.152:6792/0 conn(0x7f826c0a43c0 legacy=0x7f826c13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826a59c640 1 -- 172.21.15.94:0/4165805967 >> v1:172.21.15.152:6790/0 conn(0x7f826c0a38d0 legacy=0x7f826c0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110862615 0 0) 0x7f826c0a3250 con 0x7f826c0a43c0 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): my global_id is 4660 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8254003520 con 0x7f826c0a43c0 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493571570 0 0) 0x7f826c061ae0 con 0x7f826c0a38d0 2024-09-16T10:31:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): my global_id is 4854 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c0a3250 con 0x7f826c0a38d0 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456982602 0 0) 0x7f826c0a04d0 con 0x7f826c06c670 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 10 monclient(hunting): my global_id is 4667 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c061ae0 con 0x7f826c06c670 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.057+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2683773122 0 0) 0x7f826c0a3250 con 0x7f826c0a38d0 2024-09-16T10:31:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 >> v1:172.21.15.152:6792/0 conn(0x7f826c0a43c0 legacy=0x7f826c13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 >> v1:172.21.15.152:6791/0 conn(0x7f826c06c670 legacy=0x7f826c13bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: _finish_hunting 0 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 1 monclient: found mon.noname-g 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.152:6790/0 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0a0a00 con 0x7f826c0a38d0 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: _finish_auth 0 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: _check_auth_tickets 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f825c001b70 con 0x7f826c0a38d0 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 5 monclient: authenticate success, global_id 4854 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: dump: 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.066 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 <== mon.3 v1:172.21.15.152:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f825c003610 con 0x7f826c0a38d0 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 1 -- 172.21.15.94:0/4165805967 <== mon.3 v1:172.21.15.152:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f825c003b80 con 0x7f826c0a38d0 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 4 set_mon_vals no callback set 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:31:52.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f826959a640 10 monclient: dump: 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 10 monclient: shutdown 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 1 -- 172.21.15.94:0/4165805967 >> v1:172.21.15.152:6790/0 conn(0x7f826c0a38d0 legacy=0x7f826c0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 1 -- 172.21.15.94:0/4165805967 shutdown_connections 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 1 -- 172.21.15.94:0/4165805967 >> 172.21.15.94:0/4165805967 conn(0x7f826c0a59d0 msgr2=0x7f826c0a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 1 -- 172.21.15.94:0/4165805967 shutdown_connections 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.058+0000 7f8270c44640 1 -- 172.21.15.94:0/4165805967 wait complete. 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.059+0000 7f8270c44640 10 monclient: build_initial_monmap 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.059+0000 7f8270c44640 10 monclient: monmap: 2024-09-16T10:31:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.059+0000 7f8270c44640 1 Processor -- start 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.059+0000 7f8270c44640 1 -- start start 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.059+0000 7f8270c44640 10 monclient: init 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 10 monclient: _add_conns ranks=[7,6,4,2,1,5,0,3,8] 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 10 monclient(hunting): picked mon.noname-i con 0x7f826c06c670 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 10 monclient(hunting): picked mon.noname-d con 0x7f826c0a38d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 10 monclient(hunting): picked mon.noname-c con 0x7f826c0a43c0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c06b780 con 0x7f826c0a43c0 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c1232d0 con 0x7f826c0a38d0 2024-09-16T10:31:52.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c09eff0 con 0x7f826c06c670 2024-09-16T10:31:52.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:52.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8270c44640 10 monclient(hunting): authenticate will time out at 879.400368s 2024-09-16T10:31:52.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8269d9b640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f826c0a38d0 0x7f826c1cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:49024/0 (socket says 172.21.15.94:49024) 2024-09-16T10:31:52.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.060+0000 7f8269d9b640 1 -- 172.21.15.94:0/1733753317 learned_addr learned my addr 172.21.15.94:0/1733753317 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f826ad9d640 1 -- 172.21.15.94:0/1733753317 >> v1:172.21.15.94:6791/0 conn(0x7f826c0a43c0 legacy=0x7f826c1d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f8269d9b640 1 -- 172.21.15.94:0/1733753317 >> v1:172.21.15.94:6792/0 conn(0x7f826c0a38d0 legacy=0x7f826c1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f826a59c640 1 -- 172.21.15.94:0/1733753317 >> v1:172.21.15.152:6792/0 conn(0x7f826c06c670 legacy=0x7f826c134710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232897782 0 0) 0x7f826c06b780 con 0x7f826c0a43c0 2024-09-16T10:31:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): my global_id is 4657 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f824c003520 con 0x7f826c0a43c0 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365454222 0 0) 0x7f826c09eff0 con 0x7f826c06c670 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): my global_id is 4669 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c06b780 con 0x7f826c06c670 2024-09-16T10:31:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668357251 0 0) 0x7f826c1232d0 con 0x7f826c0a38d0 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 10 monclient(hunting): my global_id is 94182 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.062+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c09eff0 con 0x7f826c0a38d0 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 611048378 0 0) 0x7f824c003520 con 0x7f826c0a43c0 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f826c1232d0 con 0x7f826c0a43c0 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8264002030 con 0x7f826c0a43c0 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): dump: 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.079 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1951973751 0 0) 0x7f826c06b780 con 0x7f826c06c670 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f824c003520 con 0x7f826c06c670 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 99122743 0 0) 0x7f826c09eff0 con 0x7f826c0a38d0 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f826c06b780 con 0x7f826c0a38d0 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f825c003ac0 con 0x7f826c06c670 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): dump: 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.080 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.081 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8260001cb0 con 0x7f826c0a38d0 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient(hunting): dump: 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4118450264 0 0) 0x7f826c1232d0 con 0x7f826c0a43c0 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 >> v1:172.21.15.152:6792/0 conn(0x7f826c06c670 legacy=0x7f826c134710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 >> v1:172.21.15.94:6792/0 conn(0x7f826c0a38d0 legacy=0x7f826c1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient: _finish_hunting 0 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 monclient: found mon.c 2024-09-16T10:31:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c09ecc0 con 0x7f826c0a43c0 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient: _finish_auth 0 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient: _check_auth_tickets 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f8270c44640 5 monclient: authenticate success, global_id 4657 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f8270c44640 10 monclient: _renew_subs 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f8270c44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c124e40 con 0x7f826c0a43c0 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f8270c44640 10 monclient: _renew_subs 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f8270c44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c134e20 con 0x7f826c0a43c0 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8264003c40 con 0x7f826c0a43c0 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f82640040f0 con 0x7f826c0a43c0 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f8268d99640 4 set_mon_vals no callback set 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.063+0000 7f825affd640 10 monclient: dump: 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.085 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.086 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.086 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.086 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.086 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.086 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.086 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.086 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.086 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.086 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.064+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f8264004250 con 0x7f826c0a43c0 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.064+0000 7f8270c44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.064+0000 7f8270c44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.064+0000 7f8270c44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.064+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8250005040 con 0x7f826c0a43c0 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.064+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f82640461e0 con 0x7f826c0a43c0 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.064+0000 7f826a59c640 10 monclient: get_auth_request con 0x7f824c039ce0 auth_method 0 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.069+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f8264015690 con 0x7f826c0a43c0 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.069+0000 7f825affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:52.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.069+0000 7f825affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:52.193 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.193 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=574021547,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgrstat=4198604887,mon_config_key=752017177,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824888,osd_snap=2033388227,osdmap=51985567}) 2024-09-16T10:31:52.193 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=1300781486,mdsmap=3140514009,mgr=1523194813,mgr_command_descs=1973851347,mgr_metadata=955174795,mgrstat=1889778440,mon_config_key=1863606557,monmap=3140514009,osd_metadata=1666740011,osd_pg_creating=1275824887,osd_snap=2033388227,osdmap=1604446241}) 2024-09-16T10:31:52.193 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.193 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=574021547,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgrstat=4198604887,mon_config_key=752017177,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824888,osd_snap=2033388227,osdmap=51985567}) 2024-09-16T10:31:52.193 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=4239487870,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgrstat=1889778440,mon_config_key=732916039,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824887,osd_snap=2033388227,osdmap=1686626326}) 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=574021547,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgrstat=4198604887,mon_config_key=752017177,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824888,osd_snap=2033388227,osdmap=51985567}) 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=31} crc {logm=67198504,mdsmap=1531151339,mgr=195126626,mgr_command_descs=1973851347,mgr_metadata=955174795,mgrstat=1889778440,mon_config_key=338302709,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824887,osd_snap=3823785925,osdmap=701589632}) 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=574021547,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgrstat=4198604887,mon_config_key=752017177,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824888,osd_snap=2033388227,osdmap=51985567}) 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=3406002584,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgr_metadata=955174795,mgrstat=1889778440,mon_config_key=443921277,monmap=2402014165,osd_metadata=1027540971,osd_pg_creating=1275824887,osd_snap=2033388227,osdmap=359334319}) 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=574021547,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgrstat=4198604887,mon_config_key=752017177,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824888,osd_snap=2033388227,osdmap=51985567}) 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=574021547,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgr_metadata=955174795,mgrstat=1671124537,mon_config_key=209341383,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824887,osd_snap=2033388227,osdmap=349091444}) 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=574021547,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgrstat=4198604887,mon_config_key=752017177,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824888,osd_snap=2033388227,osdmap=51985567}) 2024-09-16T10:31:52.194 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=32} crc {logm=3995341988,mdsmap=1531151339,mgr=1523194813,mgr_metadata=955174795,mgrstat=1889778440,mon_config_key=338302709,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824887,osd_snap=4249038086,osdmap=459070004}) 2024-09-16T10:31:52.195 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.195 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=574021547,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgrstat=4198604887,mon_config_key=752017177,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824888,osd_snap=2033388227,osdmap=51985567}) 2024-09-16T10:31:52.195 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=2025091346,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgr_metadata=955174795,mgrstat=1889778440,mon_config_key=443921277,monmap=2402014165,osd_metadata=1027540971,osd_pg_creating=1275824887,osd_snap=2033388227,osdmap=359334319}) 2024-09-16T10:31:52.195 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.195 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=574021547,mdsmap=1531151339,mgr=1523194813,mgr_command_descs=1973851347,mgrstat=4198604887,mon_config_key=752017177,monmap=2402014165,osd_metadata=677659896,osd_pg_creating=1275824888,osd_snap=2033388227,osdmap=51985567}) 2024-09-16T10:31:52.195 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.190+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=22,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=33} crc {logm=574021547,mdsmap=1531151339,mgr=3273302482,mgr_metadata=955174795,mgrstat=423767971,mon_config_key=338302709,monmap=3140514009,osd_metadata=677659896,osd_pg_creating=1275824887,osd_snap=3823785925,osdmap=4125340821}) 2024-09-16T10:31:52.199 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.196+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.199 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.196+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=13} crc {osdmap=1678623046}) 2024-09-16T10:31:52.199 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.196+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=12} crc {osdmap=3221103669}) 2024-09-16T10:31:52.199 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.196+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.199 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.196+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=13} crc {osdmap=1678623046}) 2024-09-16T10:31:52.199 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.196+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=11} crc {osdmap=1810960158}) 2024-09-16T10:31:52.199 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.199 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=13} crc {osdmap=1678623046}) 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=12} crc {osdmap=3954892615}) 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=13} crc {osdmap=1678623046}) 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=13} crc {osdmap=1533465141}) 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=13} crc {osdmap=1678623046}) 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=12} crc {osdmap=2626382903}) 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=13} crc {osdmap=1678623046}) 2024-09-16T10:31:52.200 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-09-16T10:31:52.197+0000 7f895cb2c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=12} crc {osdmap=2626382903}) 2024-09-16T10:31:52.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.293+0000 7f8270c44640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:31:52.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.293+0000 7f8270c44640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:31:52.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.293+0000 7f8270c44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.293+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8250005870 con 0x7f826c0a43c0 2024-09-16T10:31:52.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.294+0000 7f825affd640 1 -- 172.21.15.94:0/1733753317 <== mon.4 v1:172.21.15.94:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+2712 (unknown 254746657 0 1725018639) 0x7f826401dce0 con 0x7f826c0a43c0 2024-09-16T10:31:52.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.294+0000 7f825affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:31:52.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.294+0000 7f825affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:52.296 INFO:teuthology.orchestra.run.smithi094.stdout:{"election_epoch":18,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","modified":"2024-09-16T10:28:46.257482Z","created":"2024-09-16T10:28:46.257482Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6790","nonce":0}]},"addr":"172.21.15.94:6790/0","public_addr":"172.21.15.94:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6790","nonce":0}]},"addr":"172.21.15.152:6790/0","public_addr":"172.21.15.152:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6791","nonce":0}]},"addr":"172.21.15.94:6791/0","public_addr":"172.21.15.94:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6791","nonce":0}]},"addr":"172.21.15.152:6791/0","public_addr":"172.21.15.152:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6792","nonce":0}]},"addr":"172.21.15.94:6792/0","public_addr":"172.21.15.94:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6792","nonce":0}]},"addr":"172.21.15.152:6792/0","public_addr":"172.21.15.152:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6793","nonce":0}]},"addr":"172.21.15.94:6793/0","public_addr":"172.21.15.94:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:31:52.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.296+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 >> v1:172.21.15.152:6800/2686360781 conn(0x7f824c039ce0 legacy=0x7f824c03c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.296+0000 7f8270c44640 10 monclient: shutdown 2024-09-16T10:31:52.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.296+0000 7f8270c44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:52.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.296+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 >> v1:172.21.15.94:6791/0 conn(0x7f826c0a43c0 legacy=0x7f826c1d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.296+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 shutdown_connections 2024-09-16T10:31:52.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.296+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 >> 172.21.15.94:0/1733753317 conn(0x7f826c0a59d0 msgr2=0x7f826c0a8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:52.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.296+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 shutdown_connections 2024-09-16T10:31:52.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.296+0000 7f8270c44640 1 -- 172.21.15.94:0/1733753317 wait complete. 2024-09-16T10:31:52.308 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-16T10:31:52.308 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T10:31:52.318 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T10:31:52.319 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.413+0000 7f9615012640 10 monclient: get_monmap_and_config 2024-09-16T10:31:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.413+0000 7f9615012640 10 monclient: build_initial_monmap 2024-09-16T10:31:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.413+0000 7f9615012640 10 monclient: monmap: 2024-09-16T10:31:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:52.414429+0000 2024-09-16T10:31:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:52.414429+0000 2024-09-16T10:31:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:52.416 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 1 Processor -- start 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 1 -- start start 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 10 monclient: init 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 10 monclient: _add_conns ranks=[3,2,8,4,1,7,6,5,0] 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 10 monclient(hunting): picked mon.noname-g con 0x7f96100a3840 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 10 monclient(hunting): picked mon.noname-b con 0x7f96100a17d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 10 monclient(hunting): picked mon.noname-e con 0x7f96100a4330 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9610061ae0 con 0x7f96100a17d0 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96100a3280 con 0x7f96100a3840 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961006c7d0 con 0x7f96100a4330 2024-09-16T10:31:52.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:52.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f9615012640 10 monclient(hunting): authenticate will time out at 879.754765s 2024-09-16T10:31:52.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f960e59c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f96100a17d0 0x7f961013ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41096/0 (socket says 172.21.15.94:41096) 2024-09-16T10:31:52.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f960f59e640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f96100a4330 0x7f961013f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:44082/0 (socket says 172.21.15.94:44082) 2024-09-16T10:31:52.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.414+0000 7f960e59c640 1 -- 172.21.15.94:0/3211368294 learned_addr learned my addr 172.21.15.94:0/3211368294 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960e59c640 1 -- 172.21.15.94:0/3211368294 >> v1:172.21.15.94:6790/0 conn(0x7f96100a17d0 legacy=0x7f961013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960f59e640 1 -- 172.21.15.94:0/3211368294 >> v1:172.21.15.94:6793/0 conn(0x7f96100a4330 legacy=0x7f961013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001294118 0 0) 0x7f961006c7d0 con 0x7f96100a4330 2024-09-16T10:31:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient(hunting): my global_id is 4634 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f8003520 con 0x7f96100a4330 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026086294 0 0) 0x7f9610061ae0 con 0x7f96100a17d0 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient(hunting): my global_id is 4592 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961006c7d0 con 0x7f96100a17d0 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960ed9d640 1 -- 172.21.15.94:0/3211368294 >> v1:172.21.15.152:6790/0 conn(0x7f96100a3840 legacy=0x7f96100a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261656249 0 0) 0x7f95f8003520 con 0x7f96100a4330 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 >> v1:172.21.15.152:6790/0 conn(0x7f96100a3840 legacy=0x7f96100a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 >> v1:172.21.15.94:6790/0 conn(0x7f96100a17d0 legacy=0x7f961013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: _finish_hunting 0 2024-09-16T10:31:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 monclient: found mon.noname-e 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96100a0550 con 0x7f96100a4330 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: _finish_auth 0 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: _check_auth_tickets 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9608001de0 con 0x7f96100a4330 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f9615012640 5 monclient: authenticate success, global_id 4634 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f9615012640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: dump: 2024-09-16T10:31:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.425 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f9615012640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96080020f0 con 0x7f96100a4330 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 1 -- 172.21.15.94:0/3211368294 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9608004ea0 con 0x7f96100a4330 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f9615012640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f9615012640 4 set_mon_vals no callback set 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f960dd9b640 10 monclient: dump: 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.426 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f9615012640 10 monclient: shutdown 2024-09-16T10:31:52.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.417+0000 7f9615012640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.418+0000 7f9615012640 1 -- 172.21.15.94:0/3211368294 >> v1:172.21.15.94:6793/0 conn(0x7f96100a4330 legacy=0x7f961013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.418+0000 7f9615012640 1 -- 172.21.15.94:0/3211368294 shutdown_connections 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.418+0000 7f9615012640 1 -- 172.21.15.94:0/3211368294 >> 172.21.15.94:0/3211368294 conn(0x7f96100a97b0 msgr2=0x7f96100a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.418+0000 7f9615012640 1 -- 172.21.15.94:0/3211368294 shutdown_connections 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.418+0000 7f9615012640 1 -- 172.21.15.94:0/3211368294 wait complete. 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.418+0000 7f9615012640 10 monclient: build_initial_monmap 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.418+0000 7f9615012640 10 monclient: monmap: 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 1 Processor -- start 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 1 -- start start 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 10 monclient: init 2024-09-16T10:31:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 10 monclient: _add_conns ranks=[8,4,7,3,5,1,6,2,0] 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 10 monclient(hunting): picked mon.noname-e con 0x7f96100a3840 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 10 monclient(hunting): picked mon.noname-c con 0x7f96100a4330 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 10 monclient(hunting): picked mon.noname-i con 0x7f9610137080 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9610125510 con 0x7f96100a4330 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961009f420 con 0x7f9610137080 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961009f0f0 con 0x7f96100a3840 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f9615012640 10 monclient(hunting): authenticate will time out at 879.759854s 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f960ed9d640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f96100a3840 0x7f9610136970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:44094/0 (socket says 172.21.15.94:44094) 2024-09-16T10:31:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.419+0000 7f960ed9d640 1 -- 172.21.15.94:0/2449643956 learned_addr learned my addr 172.21.15.94:0/2449643956 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f960ed9d640 1 -- 172.21.15.94:0/2449643956 >> v1:172.21.15.94:6793/0 conn(0x7f96100a3840 legacy=0x7f9610136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f960e59c640 1 -- 172.21.15.94:0/2449643956 >> v1:172.21.15.94:6791/0 conn(0x7f96100a4330 legacy=0x7f96101cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f960ed9d640 1 --1- 172.21.15.94:0/2449643956 >> v1:172.21.15.94:6793/0 conn(0x7f96100a3840 0x7f9610136970 :-1 s=OPENED pgs=134 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7f961009f0f0 auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f960f59e640 1 -- 172.21.15.94:0/2449643956 >> v1:172.21.15.152:6792/0 conn(0x7f9610137080 legacy=0x7f96101d3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440249079 0 0) 0x7f9610125510 con 0x7f96100a4330 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): my global_id is 4666 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95fc003520 con 0x7f96100a4330 2024-09-16T10:31:52.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347390875 0 0) 0x7f961009f420 con 0x7f9610137080 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): my global_id is 4678 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9610125510 con 0x7f9610137080 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4256549328 0 0) 0x7f95fc003520 con 0x7f96100a4330 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f961009f420 con 0x7f96100a4330 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9604001d10 con 0x7f96100a4330 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): dump: 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.435 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.436 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 930770622 0 0) 0x7f961009f420 con 0x7f96100a4330 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 >> v1:172.21.15.94:6793/0 conn(0x7f96100a3840 legacy=0x7f9610136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 >> v1:172.21.15.152:6792/0 conn(0x7f9610137080 legacy=0x7f96101d3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 monclient: found mon.c 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9610127080 con 0x7f96100a4330 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:52.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f9615012640 5 monclient: authenticate success, global_id 4666 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582667659 0 0) 0x7f961009f0f0 con 0x7f96100a3840 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f9615012640 10 monclient: _renew_subs 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f9615012640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9610127360 con 0x7f96100a4330 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f9615012640 10 monclient: _renew_subs 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f9615012640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96101caaa0 con 0x7f96100a4330 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9604004dd0 con 0x7f96100a4330 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9604005280 con 0x7f96100a4330 2024-09-16T10:31:52.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f95f77fe640 10 monclient: dump: 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.439 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.422+0000 7f960d59a640 4 set_mon_vals no callback set 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.423+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f96040102e0 con 0x7f96100a4330 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.423+0000 7f9615012640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.423+0000 7f9615012640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.423+0000 7f9615012640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.423+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9610067530 con 0x7f96100a4330 2024-09-16T10:31:52.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.423+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f9604045ef0 con 0x7f96100a4330 2024-09-16T10:31:52.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.423+0000 7f960ed9d640 10 monclient: get_auth_request con 0x7f95fc0390b0 auth_method 0 2024-09-16T10:31:52.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.428+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f96040153a0 con 0x7f96100a4330 2024-09-16T10:31:52.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.428+0000 7f95f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:52.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.428+0000 7f95f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:52.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.586+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f961006ba80 con 0x7f95fc0390b0 2024-09-16T10:31:52.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.588+0000 7f95f77fe640 1 -- 172.21.15.94:0/2449643956 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+54154 (unknown 2965378022 0 182268501) 0x7f961006ba80 con 0x7f95fc0390b0 2024-09-16T10:31:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:31:52.591 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-09-16T10:31:52.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.590+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 >> v1:172.21.15.152:6800/2686360781 conn(0x7f95fc0390b0 legacy=0x7f95fc03b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.590+0000 7f9615012640 10 monclient: shutdown 2024-09-16T10:31:52.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.590+0000 7f9615012640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:52.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.590+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 >> v1:172.21.15.94:6791/0 conn(0x7f96100a4330 legacy=0x7f96101cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.590+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 shutdown_connections 2024-09-16T10:31:52.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.590+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 >> 172.21.15.94:0/2449643956 conn(0x7f96100a97b0 msgr2=0x7f961009ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:52.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.590+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 shutdown_connections 2024-09-16T10:31:52.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.590+0000 7f9615012640 1 -- 172.21.15.94:0/2449643956 wait complete. 2024-09-16T10:31:52.602 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-16T10:31:51.031369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176624,"kb_used_data":2200,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054416,"statfs":{"total":579820584960,"available":579639721984,"internally_reserved":0,"allocated":2252800,"data_stored":1470364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002378"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.316550+0000","last_change":"2024-09-16T10:29:32.967697+0000","last_active":"2024-09-16T10:30:04.316550+0000","last_peered":"2024-09-16T10:30:04.316550+0000","last_clean":"2024-09-16T10:30:04.316550+0000","last_became_active":"2024-09-16T10:29:30.052739+0000","last_became_peered":"2024-09-16T10:29:30.052739+0000","last_unstale":"2024-09-16T10:30:04.316550+0000","last_undegraded":"2024-09-16T10:30:04.316550+0000","last_fullsized":"2024-09-16T10:30:04.316550+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:38:23.796765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035088200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.411813+0000","last_change":"2024-09-16T10:29:32.789258+0000","last_active":"2024-09-16T10:30:04.411813+0000","last_peered":"2024-09-16T10:30:04.411813+0000","last_clean":"2024-09-16T10:30:04.411813+0000","last_became_active":"2024-09-16T10:29:30.043984+0000","last_became_peered":"2024-09-16T10:29:30.043984+0000","last_unstale":"2024-09-16T10:30:04.411813+0000","last_undegraded":"2024-09-16T10:30:04.411813+0000","last_fullsized":"2024-09-16T10:30:04.411813+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:42.075436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028018599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.316601+0000","last_change":"2024-09-16T10:29:32.967858+0000","last_active":"2024-09-16T10:30:04.316601+0000","last_peered":"2024-09-16T10:30:04.316601+0000","last_clean":"2024-09-16T10:30:04.316601+0000","last_became_active":"2024-09-16T10:29:30.043883+0000","last_became_peered":"2024-09-16T10:29:30.043883+0000","last_unstale":"2024-09-16T10:30:04.316601+0000","last_undegraded":"2024-09-16T10:30:04.316601+0000","last_fullsized":"2024-09-16T10:30:04.316601+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:31.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048663500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.412640+0000","last_change":"2024-09-16T10:29:32.789216+0000","last_active":"2024-09-16T10:30:04.412640+0000","last_peered":"2024-09-16T10:30:04.412640+0000","last_clean":"2024-09-16T10:30:04.412640+0000","last_became_active":"2024-09-16T10:29:30.039372+0000","last_became_peered":"2024-09-16T10:29:30.039372+0000","last_unstale":"2024-09-16T10:30:04.412640+0000","last_undegraded":"2024-09-16T10:30:04.412640+0000","last_fullsized":"2024-09-16T10:30:04.412640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:15.482545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027726900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.412098+0000","last_change":"2024-09-16T10:29:32.794417+0000","last_active":"2024-09-16T10:30:04.412098+0000","last_peered":"2024-09-16T10:30:04.412098+0000","last_clean":"2024-09-16T10:30:04.412098+0000","last_became_active":"2024-09-16T10:29:30.039308+0000","last_became_peered":"2024-09-16T10:29:30.039308+0000","last_unstale":"2024-09-16T10:30:04.412098+0000","last_undegraded":"2024-09-16T10:30:04.412098+0000","last_fullsized":"2024-09-16T10:30:04.412098+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:04.076079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076855000000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.314171+0000","last_change":"2024-09-16T10:29:32.784986+0000","last_active":"2024-09-16T10:30:04.314171+0000","last_peered":"2024-09-16T10:30:04.314171+0000","last_clean":"2024-09-16T10:30:04.314171+0000","last_became_active":"2024-09-16T10:29:30.039431+0000","last_became_peered":"2024-09-16T10:29:30.039431+0000","last_unstale":"2024-09-16T10:30:04.314171+0000","last_undegraded":"2024-09-16T10:30:04.314171+0000","last_fullsized":"2024-09-16T10:30:04.314171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:20.175515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041305900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.314672+0000","last_change":"2024-09-16T10:29:32.967651+0000","last_active":"2024-09-16T10:30:04.314672+0000","last_peered":"2024-09-16T10:30:04.314672+0000","last_clean":"2024-09-16T10:30:04.314672+0000","last_became_active":"2024-09-16T10:29:30.043748+0000","last_became_peered":"2024-09-16T10:29:30.043748+0000","last_unstale":"2024-09-16T10:30:04.314672+0000","last_undegraded":"2024-09-16T10:30:04.314672+0000","last_fullsized":"2024-09-16T10:30:04.314672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:37:47.716642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349687,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.411572+0000","last_change":"2024-09-16T10:29:32.794058+0000","last_active":"2024-09-16T10:30:04.411572+0000","last_peered":"2024-09-16T10:30:04.411572+0000","last_clean":"2024-09-16T10:30:04.411572+0000","last_became_active":"2024-09-16T10:29:30.038374+0000","last_became_peered":"2024-09-16T10:29:30.038374+0000","last_unstale":"2024-09-16T10:30:04.411572+0000","last_undegraded":"2024-09-16T10:30:04.411572+0000","last_fullsized":"2024-09-16T10:30:04.411572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:17.735950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040381400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.315003+0000","last_change":"2024-09-16T10:29:29.005269+0000","last_active":"2024-09-16T10:30:04.315003+0000","last_peered":"2024-09-16T10:30:04.315003+0000","last_clean":"2024-09-16T10:30:04.315003+0000","last_became_active":"2024-09-16T10:29:29.004894+0000","last_became_peered":"2024-09-16T10:29:29.004894+0000","last_unstale":"2024-09-16T10:30:04.315003+0000","last_undegraded":"2024-09-16T10:30:04.315003+0000","last_fullsized":"2024-09-16T10:30:04.315003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.994140+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:40.518658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.502,"15min":0.49199999999999999},"min":{"1min":0.39400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66300000000000003,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.47599999999999998,"15min":0.46700000000000003},"min":{"1min":0.40799999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.35599999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.5},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.503,"15min":0.498},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.437}]},{"osd":3,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36599999999999999,"15min":0.36499999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.51000000000000001,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.38800000000000001},"min":{"1min":0.312,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.32000000000000001}]},{"osd":4,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.32300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.38},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.39300000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.308,"15min":0.308},"min":{"1min":0.23200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.39200000000000002,"5min":0.439,"15min":0.439},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.442,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.38200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.34300000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42399999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.34599999999999997,"15min":0.34799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.36399999999999999}]},{"osd":3,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.504,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.502},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.53800000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640286,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.627,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.443},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.376,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32300000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.50900000000000001,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63700000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.40100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.40100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.626,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65300000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.55200000000000005,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.311},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41899999999999998,"15min":0.41999999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54200000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.438,"15min":0.437},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.35099999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.505,"15min":0.495},"min":{"1min":0.41099999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68400000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.42299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.55400000000000005}]},{"osd":1,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.48099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68000000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.438},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.46200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.45300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53500000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.501}]},{"osd":4,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.32100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.29399999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.376},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.52000000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640286,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.375,"15min":0.37},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54900000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34399999999999997},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.497,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.39000000000000001}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55700000000000005}]},{"osd":4,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.41099999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78400000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56899999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640286,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.32700000000000001},"min":{"1min":0.26800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.44600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.315},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.375,"15min":0.379},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.41999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.26500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.378,"5min":0.38700000000000001,"15min":0.38900000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.48899999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.747,"15min":0.78200000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68899999999999995,"5min":5.032,"15min":5.032},"last":0.64400000000000002}]},{"osd":4,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67600000000000005,"5min":4.9240000000000004,"15min":4.9240000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.75700000000000001,"15min":0.79000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.71999999999999997,"5min":4.9800000000000004,"15min":4.9800000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.753,"5min":5.0030000000000001,"15min":5.0030000000000001},"last":0.63100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:52.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:52.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.699+0000 7ff582e9c640 10 monclient: get_monmap_and_config 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.699+0000 7ff582e9c640 10 monclient: build_initial_monmap 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.699+0000 7ff582e9c640 10 monclient: monmap: 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:52.700113+0000 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:52.700113+0000 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:52.702 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:52.702 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:52.702 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:52.702 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:52.702 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:52.702 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:52.702 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 1 Processor -- start 2024-09-16T10:31:52.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 1 -- start start 2024-09-16T10:31:52.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 10 monclient: init 2024-09-16T10:31:52.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:52.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 10 monclient: _add_conns ranks=[6,2,7,3,0,8,5,4,1] 2024-09-16T10:31:52.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 10 monclient(hunting): picked mon.noname-d con 0x7ff57c0a3840 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 10 monclient(hunting): picked mon.noname-b con 0x7ff57c0a17d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 10 monclient(hunting): picked mon.noname-i con 0x7ff57c0a4330 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c061ae0 con 0x7ff57c0a17d0 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c0a3280 con 0x7ff57c0a3840 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c06c7d0 con 0x7ff57c0a4330 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff582e9c640 10 monclient(hunting): authenticate will time out at 880.040596s 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff580c38640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7ff57c0a3840 0x7ff57c0a3c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:49036/0 (socket says 172.21.15.94:49036) 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff580c38640 1 -- 172.21.15.94:0/1596175889 learned_addr learned my addr 172.21.15.94:0/1596175889 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:52.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.700+0000 7ff57bfff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7ff57c0a17d0 0x7ff57c13ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41112/0 (socket says 172.21.15.94:41112) 2024-09-16T10:31:52.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.702+0000 7ff580c38640 1 -- 172.21.15.94:0/1596175889 >> v1:172.21.15.94:6792/0 conn(0x7ff57c0a3840 legacy=0x7ff57c0a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.702+0000 7ff57bfff640 1 -- 172.21.15.94:0/1596175889 >> v1:172.21.15.94:6790/0 conn(0x7ff57c0a17d0 legacy=0x7ff57c13ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308743149 0 0) 0x7ff57c0a3280 con 0x7ff57c0a3840 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient(hunting): my global_id is 94191 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff568003520 con 0x7ff57c0a3840 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302248338 0 0) 0x7ff57c061ae0 con 0x7ff57c0a17d0 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient(hunting): my global_id is 4601 2024-09-16T10:31:52.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff57c0a3280 con 0x7ff57c0a17d0 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff581439640 1 -- 172.21.15.94:0/1596175889 >> v1:172.21.15.152:6792/0 conn(0x7ff57c0a4330 legacy=0x7ff57c13f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 86181581 0 0) 0x7ff57c0a3280 con 0x7ff57c0a17d0 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 >> v1:172.21.15.152:6792/0 conn(0x7ff57c0a4330 legacy=0x7ff57c13f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 >> v1:172.21.15.94:6792/0 conn(0x7ff57c0a3840 legacy=0x7ff57c0a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 monclient: found mon.noname-b 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c0a0550 con 0x7ff57c0a17d0 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff56c001de0 con 0x7ff57c0a17d0 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff582e9c640 5 monclient: authenticate success, global_id 4601 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff582e9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: dump: 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff582e9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff56c0020f0 con 0x7ff57c0a17d0 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 1 -- 172.21.15.94:0/1596175889 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff56c004ea0 con 0x7ff57c0a17d0 2024-09-16T10:31:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff582e9c640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff582e9c640 4 set_mon_vals no callback set 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff57b7fe640 10 monclient: dump: 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff582e9c640 10 monclient: shutdown 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff582e9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.703+0000 7ff582e9c640 1 -- 172.21.15.94:0/1596175889 >> v1:172.21.15.94:6790/0 conn(0x7ff57c0a17d0 legacy=0x7ff57c13ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.704+0000 7ff582e9c640 1 -- 172.21.15.94:0/1596175889 shutdown_connections 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.704+0000 7ff582e9c640 1 -- 172.21.15.94:0/1596175889 >> 172.21.15.94:0/1596175889 conn(0x7ff57c0a97b0 msgr2=0x7ff57c0a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.704+0000 7ff582e9c640 1 -- 172.21.15.94:0/1596175889 shutdown_connections 2024-09-16T10:31:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.704+0000 7ff582e9c640 1 -- 172.21.15.94:0/1596175889 wait complete. 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.704+0000 7ff582e9c640 10 monclient: build_initial_monmap 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.704+0000 7ff582e9c640 10 monclient: monmap: 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.704+0000 7ff582e9c640 1 Processor -- start 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 1 -- start start 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 10 monclient: init 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 10 monclient: _add_conns ranks=[4,6,5,1,2,0,8,7,3] 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 10 monclient(hunting): picked mon.noname-c con 0x7ff57c0a3840 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 10 monclient(hunting): picked mon.noname-d con 0x7ff57c0a4330 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:52.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 10 monclient(hunting): picked mon.noname-h con 0x7ff57c137160 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c126170 con 0x7ff57c0a3840 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c1255f0 con 0x7ff57c137160 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c09f290 con 0x7ff57c0a4330 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff582e9c640 10 monclient(hunting): authenticate will time out at 880.045717s 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff57bfff640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7ff57c0a4330 0x7ff57c1cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:49040/0 (socket says 172.21.15.94:49040) 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff580c38640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7ff57c0a3840 0x7ff57c136a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:34054/0 (socket says 172.21.15.94:34054) 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.705+0000 7ff57bfff640 1 -- 172.21.15.94:0/3481012210 learned_addr learned my addr 172.21.15.94:0/3481012210 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.707+0000 7ff580c38640 1 -- 172.21.15.94:0/3481012210 >> v1:172.21.15.94:6791/0 conn(0x7ff57c0a3840 legacy=0x7ff57c136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.707+0000 7ff57bfff640 1 -- 172.21.15.94:0/3481012210 >> v1:172.21.15.94:6792/0 conn(0x7ff57c0a4330 legacy=0x7ff57c1cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.707+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181664045 0 0) 0x7ff57c126170 con 0x7ff57c0a3840 2024-09-16T10:31:52.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): my global_id is 4675 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff55c003520 con 0x7ff57c0a3840 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427645699 0 0) 0x7ff57c09f290 con 0x7ff57c0a4330 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): my global_id is 94200 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff57c126170 con 0x7ff57c0a4330 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff581439640 1 -- 172.21.15.94:0/3481012210 >> v1:172.21.15.152:6791/0 conn(0x7ff57c137160 legacy=0x7ff57c1d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3820056264 0 0) 0x7ff55c003520 con 0x7ff57c0a3840 2024-09-16T10:31:52.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff57c09f290 con 0x7ff57c0a3840 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff564001320 con 0x7ff57c0a3840 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): dump: 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.722 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4284495003 0 0) 0x7ff57c126170 con 0x7ff57c0a4330 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff55c003520 con 0x7ff57c0a4330 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587443501 0 0) 0x7ff57c1255f0 con 0x7ff57c137160 2024-09-16T10:31:52.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): my global_id is 4676 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff57c126170 con 0x7ff57c137160 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff56c004ea0 con 0x7ff57c0a4330 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient(hunting): dump: 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.724 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2612405457 0 0) 0x7ff57c09f290 con 0x7ff57c0a3840 2024-09-16T10:31:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 >> v1:172.21.15.94:6792/0 conn(0x7ff57c0a4330 legacy=0x7ff57c1cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 >> v1:172.21.15.152:6791/0 conn(0x7ff57c137160 legacy=0x7ff57c1d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient: _finish_hunting 0 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 monclient: found mon.c 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c09ef60 con 0x7ff57c0a3840 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient: _finish_auth 0 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient: _check_auth_tickets 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff582e9c640 5 monclient: authenticate success, global_id 4675 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff582e9c640 10 monclient: _renew_subs 2024-09-16T10:31:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff582e9c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c126e80 con 0x7ff57c0a3840 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff582e9c640 10 monclient: _renew_subs 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff582e9c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57c1250e0 con 0x7ff57c0a3840 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff564001ac0 con 0x7ff57c0a3840 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7ff564005110 con 0x7ff57c0a3840 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff5797fa640 10 monclient: dump: 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.728 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.708+0000 7ff57affd640 4 set_mon_vals no callback set 2024-09-16T10:31:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.709+0000 7ff582e9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.709+0000 7ff582e9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.709+0000 7ff582e9c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.709+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7ff564010170 con 0x7ff57c0a3840 2024-09-16T10:31:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.709+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57c067530 con 0x7ff57c0a3840 2024-09-16T10:31:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.709+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7ff564045ef0 con 0x7ff57c0a3840 2024-09-16T10:31:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.710+0000 7ff57bfff640 10 monclient: get_auth_request con 0x7ff55c0399c0 auth_method 0 2024-09-16T10:31:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.713+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7ff5640153a0 con 0x7ff57c0a3840 2024-09-16T10:31:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.713+0000 7ff5797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.713+0000 7ff5797fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:52.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.871+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff57c0a2fc0 con 0x7ff55c0399c0 2024-09-16T10:31:52.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.873+0000 7ff5797fa640 1 -- 172.21.15.94:0/3481012210 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+54154 (unknown 2965378022 0 182268501) 0x7ff57c0a2fc0 con 0x7ff55c0399c0 2024-09-16T10:31:52.875 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:31:52.875 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-09-16T10:31:52.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.875+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 >> v1:172.21.15.152:6800/2686360781 conn(0x7ff55c0399c0 legacy=0x7ff55c03be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.875+0000 7ff582e9c640 10 monclient: shutdown 2024-09-16T10:31:52.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.875+0000 7ff582e9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:52.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.875+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 >> v1:172.21.15.94:6791/0 conn(0x7ff57c0a3840 legacy=0x7ff57c136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.875+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 shutdown_connections 2024-09-16T10:31:52.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.875+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 >> 172.21.15.94:0/3481012210 conn(0x7ff57c0a97b0 msgr2=0x7ff57c0a7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:52.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.875+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 shutdown_connections 2024-09-16T10:31:52.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.875+0000 7ff582e9c640 1 -- 172.21.15.94:0/3481012210 wait complete. 2024-09-16T10:31:52.888 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-16T10:31:51.031369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176624,"kb_used_data":2200,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054416,"statfs":{"total":579820584960,"available":579639721984,"internally_reserved":0,"allocated":2252800,"data_stored":1470364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002378"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.316550+0000","last_change":"2024-09-16T10:29:32.967697+0000","last_active":"2024-09-16T10:30:04.316550+0000","last_peered":"2024-09-16T10:30:04.316550+0000","last_clean":"2024-09-16T10:30:04.316550+0000","last_became_active":"2024-09-16T10:29:30.052739+0000","last_became_peered":"2024-09-16T10:29:30.052739+0000","last_unstale":"2024-09-16T10:30:04.316550+0000","last_undegraded":"2024-09-16T10:30:04.316550+0000","last_fullsized":"2024-09-16T10:30:04.316550+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:38:23.796765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035088200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.411813+0000","last_change":"2024-09-16T10:29:32.789258+0000","last_active":"2024-09-16T10:30:04.411813+0000","last_peered":"2024-09-16T10:30:04.411813+0000","last_clean":"2024-09-16T10:30:04.411813+0000","last_became_active":"2024-09-16T10:29:30.043984+0000","last_became_peered":"2024-09-16T10:29:30.043984+0000","last_unstale":"2024-09-16T10:30:04.411813+0000","last_undegraded":"2024-09-16T10:30:04.411813+0000","last_fullsized":"2024-09-16T10:30:04.411813+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:42.075436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028018599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.316601+0000","last_change":"2024-09-16T10:29:32.967858+0000","last_active":"2024-09-16T10:30:04.316601+0000","last_peered":"2024-09-16T10:30:04.316601+0000","last_clean":"2024-09-16T10:30:04.316601+0000","last_became_active":"2024-09-16T10:29:30.043883+0000","last_became_peered":"2024-09-16T10:29:30.043883+0000","last_unstale":"2024-09-16T10:30:04.316601+0000","last_undegraded":"2024-09-16T10:30:04.316601+0000","last_fullsized":"2024-09-16T10:30:04.316601+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:31.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048663500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.412640+0000","last_change":"2024-09-16T10:29:32.789216+0000","last_active":"2024-09-16T10:30:04.412640+0000","last_peered":"2024-09-16T10:30:04.412640+0000","last_clean":"2024-09-16T10:30:04.412640+0000","last_became_active":"2024-09-16T10:29:30.039372+0000","last_became_peered":"2024-09-16T10:29:30.039372+0000","last_unstale":"2024-09-16T10:30:04.412640+0000","last_undegraded":"2024-09-16T10:30:04.412640+0000","last_fullsized":"2024-09-16T10:30:04.412640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:15.482545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027726900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.412098+0000","last_change":"2024-09-16T10:29:32.794417+0000","last_active":"2024-09-16T10:30:04.412098+0000","last_peered":"2024-09-16T10:30:04.412098+0000","last_clean":"2024-09-16T10:30:04.412098+0000","last_became_active":"2024-09-16T10:29:30.039308+0000","last_became_peered":"2024-09-16T10:29:30.039308+0000","last_unstale":"2024-09-16T10:30:04.412098+0000","last_undegraded":"2024-09-16T10:30:04.412098+0000","last_fullsized":"2024-09-16T10:30:04.412098+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:04.076079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076855000000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.314171+0000","last_change":"2024-09-16T10:29:32.784986+0000","last_active":"2024-09-16T10:30:04.314171+0000","last_peered":"2024-09-16T10:30:04.314171+0000","last_clean":"2024-09-16T10:30:04.314171+0000","last_became_active":"2024-09-16T10:29:30.039431+0000","last_became_peered":"2024-09-16T10:29:30.039431+0000","last_unstale":"2024-09-16T10:30:04.314171+0000","last_undegraded":"2024-09-16T10:30:04.314171+0000","last_fullsized":"2024-09-16T10:30:04.314171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:20.175515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041305900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.314672+0000","last_change":"2024-09-16T10:29:32.967651+0000","last_active":"2024-09-16T10:30:04.314672+0000","last_peered":"2024-09-16T10:30:04.314672+0000","last_clean":"2024-09-16T10:30:04.314672+0000","last_became_active":"2024-09-16T10:29:30.043748+0000","last_became_peered":"2024-09-16T10:29:30.043748+0000","last_unstale":"2024-09-16T10:30:04.314672+0000","last_undegraded":"2024-09-16T10:30:04.314672+0000","last_fullsized":"2024-09-16T10:30:04.314672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:37:47.716642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349687,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.411572+0000","last_change":"2024-09-16T10:29:32.794058+0000","last_active":"2024-09-16T10:30:04.411572+0000","last_peered":"2024-09-16T10:30:04.411572+0000","last_clean":"2024-09-16T10:30:04.411572+0000","last_became_active":"2024-09-16T10:29:30.038374+0000","last_became_peered":"2024-09-16T10:29:30.038374+0000","last_unstale":"2024-09-16T10:30:04.411572+0000","last_undegraded":"2024-09-16T10:30:04.411572+0000","last_fullsized":"2024-09-16T10:30:04.411572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:17.735950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040381400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.315003+0000","last_change":"2024-09-16T10:29:29.005269+0000","last_active":"2024-09-16T10:30:04.315003+0000","last_peered":"2024-09-16T10:30:04.315003+0000","last_clean":"2024-09-16T10:30:04.315003+0000","last_became_active":"2024-09-16T10:29:29.004894+0000","last_became_peered":"2024-09-16T10:29:29.004894+0000","last_unstale":"2024-09-16T10:30:04.315003+0000","last_undegraded":"2024-09-16T10:30:04.315003+0000","last_fullsized":"2024-09-16T10:30:04.315003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.994140+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:40.518658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.502,"15min":0.49199999999999999},"min":{"1min":0.39400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66300000000000003,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.47599999999999998,"15min":0.46700000000000003},"min":{"1min":0.40799999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.35599999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.5},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.503,"15min":0.498},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.437}]},{"osd":3,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36599999999999999,"15min":0.36499999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.51000000000000001,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.38800000000000001},"min":{"1min":0.312,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.32000000000000001}]},{"osd":4,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.32300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.38},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.39300000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.308,"15min":0.308},"min":{"1min":0.23200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.39200000000000002,"5min":0.439,"15min":0.439},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.442,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.38200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.34300000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42399999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.34599999999999997,"15min":0.34799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.36399999999999999}]},{"osd":3,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.504,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.502},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.53800000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640286,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.627,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.443},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.376,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32300000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.50900000000000001,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63700000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.40100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.40100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.626,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65300000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.55200000000000005,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.311},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41899999999999998,"15min":0.41999999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54200000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.438,"15min":0.437},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.35099999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.505,"15min":0.495},"min":{"1min":0.41099999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68400000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.42299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.55400000000000005}]},{"osd":1,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.48099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68000000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.438},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.46200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.45300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53500000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.501}]},{"osd":4,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.32100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.29399999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.376},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.52000000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640286,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.375,"15min":0.37},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54900000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34399999999999997},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.497,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.39000000000000001}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55700000000000005}]},{"osd":4,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.41099999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78400000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56899999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640286,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.32700000000000001},"min":{"1min":0.26800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.44600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.315},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.375,"15min":0.379},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.41999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.26500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.378,"5min":0.38700000000000001,"15min":0.38900000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.48899999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.747,"15min":0.78200000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68899999999999995,"5min":5.032,"15min":5.032},"last":0.64400000000000002}]},{"osd":4,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67600000000000005,"5min":4.9240000000000004,"15min":4.9240000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.75700000000000001,"15min":0.79000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.71999999999999997,"5min":4.9800000000000004,"15min":4.9800000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.753,"5min":5.0030000000000001,"15min":5.0030000000000001},"last":0.63100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:52.890 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T10:31:52.890 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.985+0000 7f0e6603a640 10 monclient: get_monmap_and_config 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.985+0000 7f0e6603a640 10 monclient: build_initial_monmap 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.985+0000 7f0e6603a640 10 monclient: monmap: 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:52.986397+0000 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:52.986397+0000 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:52.988 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:52.988 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:52.988 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:52.988 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:52.988 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:52.988 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:52.988 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:52.988 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:52.988 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 1 Processor -- start 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 1 -- start start 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 10 monclient: init 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 10 monclient: _add_conns ranks=[7,5,2,8,4,0,6,1,3] 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 10 monclient(hunting): picked mon.noname-i con 0x7f0e600a3840 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 10 monclient(hunting): picked mon.noname-h con 0x7f0e600a17d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 10 monclient(hunting): picked mon.noname-b con 0x7f0e600a4330 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e60061ae0 con 0x7f0e600a4330 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e600a3280 con 0x7f0e600a17d0 2024-09-16T10:31:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6006c7d0 con 0x7f0e600a3840 2024-09-16T10:31:52.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:52.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e6603a640 10 monclient(hunting): authenticate will time out at 880.326824s 2024-09-16T10:31:52.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e5ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f0e600a4330 0x7f0e6013f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41118/0 (socket says 172.21.15.94:41118) 2024-09-16T10:31:52.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.986+0000 7f0e5ffff640 1 -- 172.21.15.94:0/2062195741 learned_addr learned my addr 172.21.15.94:0/2062195741 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5ffff640 1 -- 172.21.15.94:0/2062195741 >> v1:172.21.15.94:6790/0 conn(0x7f0e600a4330 legacy=0x7f0e6013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 -- 172.21.15.94:0/2062195741 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122245990 0 0) 0x7f0e60061ae0 con 0x7f0e600a4330 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient(hunting): my global_id is 4610 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 -- 172.21.15.94:0/2062195741 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e48003520 con 0x7f0e600a4330 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 -- 172.21.15.94:0/2062195741 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439227434 0 0) 0x7f0e48003520 con 0x7f0e600a4330 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5f7fe640 1 -- 172.21.15.94:0/2062195741 >> v1:172.21.15.152:6792/0 conn(0x7f0e600a3840 legacy=0x7f0e600a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 -- 172.21.15.94:0/2062195741 >> v1:172.21.15.152:6792/0 conn(0x7f0e600a3840 legacy=0x7f0e600a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5effd640 1 -- 172.21.15.94:0/2062195741 >> v1:172.21.15.152:6791/0 conn(0x7f0e600a17d0 legacy=0x7f0e6013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 -- 172.21.15.94:0/2062195741 >> v1:172.21.15.152:6791/0 conn(0x7f0e600a17d0 legacy=0x7f0e6013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 monclient: found mon.noname-b 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 -- 172.21.15.94:0/2062195741 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e600a0550 con 0x7f0e600a4330 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 -- 172.21.15.94:0/2062195741 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0e58001ea0 con 0x7f0e600a4330 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e6603a640 5 monclient: authenticate success, global_id 4610 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e6603a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:52.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: dump: 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e6603a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 -- 172.21.15.94:0/2062195741 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e58004c20 con 0x7f0e600a4330 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 1 -- 172.21.15.94:0/2062195741 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0e58005110 con 0x7f0e600a4330 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:52.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e5e7fc640 10 monclient: dump: 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:52.998 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e6603a640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.989+0000 7f0e6603a640 4 set_mon_vals no callback set 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.990+0000 7f0e6603a640 10 monclient: shutdown 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.990+0000 7f0e6603a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.990+0000 7f0e6603a640 1 -- 172.21.15.94:0/2062195741 >> v1:172.21.15.94:6790/0 conn(0x7f0e600a4330 legacy=0x7f0e6013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.990+0000 7f0e6603a640 1 -- 172.21.15.94:0/2062195741 shutdown_connections 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.990+0000 7f0e6603a640 1 -- 172.21.15.94:0/2062195741 >> 172.21.15.94:0/2062195741 conn(0x7f0e600a97b0 msgr2=0x7f0e600a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.990+0000 7f0e6603a640 1 -- 172.21.15.94:0/2062195741 shutdown_connections 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.990+0000 7f0e6603a640 1 -- 172.21.15.94:0/2062195741 wait complete. 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.990+0000 7f0e6603a640 10 monclient: build_initial_monmap 2024-09-16T10:31:52.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.990+0000 7f0e6603a640 10 monclient: monmap: 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:53.000 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 1 Processor -- start 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 1 -- start start 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 10 monclient: init 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 10 monclient: _add_conns ranks=[7,0,6,5,3,8,2,4,1] 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 10 monclient(hunting): picked mon.noname-i con 0x7f0e600a3840 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 10 monclient(hunting): picked mon.noname-a con 0x7f0e600a4330 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 10 monclient(hunting): picked mon.noname-d con 0x7f0e601370c0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6009f320 con 0x7f0e600a4330 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e601273a0 con 0x7f0e601370c0 2024-09-16T10:31:53.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6009eff0 con 0x7f0e600a3840 2024-09-16T10:31:53.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:53.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e6603a640 10 monclient(hunting): authenticate will time out at 880.331670s 2024-09-16T10:31:53.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e5effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0e600a4330 0x7f0e601cfdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53468/0 (socket says 172.21.15.94:53468) 2024-09-16T10:31:53.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.991+0000 7f0e5effd640 1 -- 172.21.15.94:0/3857567085 learned_addr learned my addr 172.21.15.94:0/3857567085 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:53.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.993+0000 7f0e5ffff640 1 -- 172.21.15.94:0/3857567085 >> v1:172.21.15.94:6792/0 conn(0x7f0e601370c0 legacy=0x7f0e601d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:53.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.993+0000 7f0e5effd640 1 -- 172.21.15.94:0/3857567085 >> v1:172.21.15.94:6789/0 conn(0x7f0e600a4330 legacy=0x7f0e601cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:53.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319318611 0 0) 0x7f0e601273a0 con 0x7f0e601370c0 2024-09-16T10:31:53.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:53.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:53.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:53.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): my global_id is 94209 2024-09-16T10:31:53.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e4c003520 con 0x7f0e601370c0 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e5f7fe640 1 -- 172.21.15.94:0/3857567085 >> v1:172.21.15.152:6792/0 conn(0x7f0e600a3840 legacy=0x7f0e601369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596188195 0 0) 0x7f0e6009f320 con 0x7f0e600a4330 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): my global_id is 4581 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e601273a0 con 0x7f0e600a4330 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 262249974 0 0) 0x7f0e4c003520 con 0x7f0e601370c0 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e6009f320 con 0x7f0e601370c0 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0e580050d0 con 0x7f0e601370c0 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): dump: 2024-09-16T10:31:53.006 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:53.007 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:53.008 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:53.008 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2453864435 0 0) 0x7f0e601273a0 con 0x7f0e600a4330 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e4c003520 con 0x7f0e600a4330 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779830752 0 0) 0x7f0e6009eff0 con 0x7f0e600a3840 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): my global_id is 4696 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e601273a0 con 0x7f0e600a3840 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0e54001c40 con 0x7f0e600a4330 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient(hunting): dump: 2024-09-16T10:31:53.009 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:53.010 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 789531408 0 0) 0x7f0e6009f320 con 0x7f0e601370c0 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 >> v1:172.21.15.152:6792/0 conn(0x7f0e600a3840 legacy=0x7f0e601369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 >> v1:172.21.15.94:6789/0 conn(0x7f0e600a4330 legacy=0x7f0e601cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient: _finish_hunting 0 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 monclient: found mon.d 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6006b980 con 0x7f0e601370c0 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient: _finish_auth 0 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient: _check_auth_tickets 2024-09-16T10:31:53.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e6603a640 5 monclient: authenticate success, global_id 94209 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e6603a640 10 monclient: _renew_subs 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e6603a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e60126de0 con 0x7f0e601370c0 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e6603a640 10 monclient: _renew_subs 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e6603a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e60125720 con 0x7f0e601370c0 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e58001e80 con 0x7f0e601370c0 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f0e58005910 con 0x7f0e601370c0 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:53.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e47fff640 10 monclient: dump: 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:53.013 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.994+0000 7f0e5dffb640 4 set_mon_vals no callback set 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.995+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f0e58010970 con 0x7f0e601370c0 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.995+0000 7f0e6603a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.995+0000 7f0e6603a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.995+0000 7f0e6603a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.995+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e60067530 con 0x7f0e601370c0 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.995+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f0e58046490 con 0x7f0e601370c0 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:52.995+0000 7f0e5f7fe640 10 monclient: get_auth_request con 0x7f0e4c0399b0 auth_method 0 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.000+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f0e58015a70 con 0x7f0e601370c0 2024-09-16T10:31:53.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.000+0000 7f0e47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:53.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.000+0000 7f0e47fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:53.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.160+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e60125570 con 0x7f0e4c0399b0 2024-09-16T10:31:53.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.162+0000 7f0e47fff640 1 -- 172.21.15.94:0/3857567085 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+54111 (unknown 2965378022 0 1468396831) 0x7f0e60125570 con 0x7f0e4c0399b0 2024-09-16T10:31:53.164 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:31:53.164 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-09-16T10:31:53.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.164+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 >> v1:172.21.15.152:6800/2686360781 conn(0x7f0e4c0399b0 legacy=0x7f0e4c03be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.164+0000 7f0e6603a640 10 monclient: shutdown 2024-09-16T10:31:53.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.164+0000 7f0e6603a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:53.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.164+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 >> v1:172.21.15.94:6792/0 conn(0x7f0e601370c0 legacy=0x7f0e601d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.164+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 shutdown_connections 2024-09-16T10:31:53.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.164+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 >> 172.21.15.94:0/3857567085 conn(0x7f0e600a97b0 msgr2=0x7f0e600a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:53.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.164+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 shutdown_connections 2024-09-16T10:31:53.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.164+0000 7f0e6603a640 1 -- 172.21.15.94:0/3857567085 wait complete. 2024-09-16T10:31:53.176 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-16T10:31:53.031645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176624,"kb_used_data":2200,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054416,"statfs":{"total":579820584960,"available":579639721984,"internally_reserved":0,"allocated":2252800,"data_stored":1470364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002331"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.316550+0000","last_change":"2024-09-16T10:29:32.967697+0000","last_active":"2024-09-16T10:30:04.316550+0000","last_peered":"2024-09-16T10:30:04.316550+0000","last_clean":"2024-09-16T10:30:04.316550+0000","last_became_active":"2024-09-16T10:29:30.052739+0000","last_became_peered":"2024-09-16T10:29:30.052739+0000","last_unstale":"2024-09-16T10:30:04.316550+0000","last_undegraded":"2024-09-16T10:30:04.316550+0000","last_fullsized":"2024-09-16T10:30:04.316550+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:38:23.796765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035088200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.411813+0000","last_change":"2024-09-16T10:29:32.789258+0000","last_active":"2024-09-16T10:30:04.411813+0000","last_peered":"2024-09-16T10:30:04.411813+0000","last_clean":"2024-09-16T10:30:04.411813+0000","last_became_active":"2024-09-16T10:29:30.043984+0000","last_became_peered":"2024-09-16T10:29:30.043984+0000","last_unstale":"2024-09-16T10:30:04.411813+0000","last_undegraded":"2024-09-16T10:30:04.411813+0000","last_fullsized":"2024-09-16T10:30:04.411813+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:42.075436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028018599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.316601+0000","last_change":"2024-09-16T10:29:32.967858+0000","last_active":"2024-09-16T10:30:04.316601+0000","last_peered":"2024-09-16T10:30:04.316601+0000","last_clean":"2024-09-16T10:30:04.316601+0000","last_became_active":"2024-09-16T10:29:30.043883+0000","last_became_peered":"2024-09-16T10:29:30.043883+0000","last_unstale":"2024-09-16T10:30:04.316601+0000","last_undegraded":"2024-09-16T10:30:04.316601+0000","last_fullsized":"2024-09-16T10:30:04.316601+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:31.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048663500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.412640+0000","last_change":"2024-09-16T10:29:32.789216+0000","last_active":"2024-09-16T10:30:04.412640+0000","last_peered":"2024-09-16T10:30:04.412640+0000","last_clean":"2024-09-16T10:30:04.412640+0000","last_became_active":"2024-09-16T10:29:30.039372+0000","last_became_peered":"2024-09-16T10:29:30.039372+0000","last_unstale":"2024-09-16T10:30:04.412640+0000","last_undegraded":"2024-09-16T10:30:04.412640+0000","last_fullsized":"2024-09-16T10:30:04.412640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:15.482545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027726900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.412098+0000","last_change":"2024-09-16T10:29:32.794417+0000","last_active":"2024-09-16T10:30:04.412098+0000","last_peered":"2024-09-16T10:30:04.412098+0000","last_clean":"2024-09-16T10:30:04.412098+0000","last_became_active":"2024-09-16T10:29:30.039308+0000","last_became_peered":"2024-09-16T10:29:30.039308+0000","last_unstale":"2024-09-16T10:30:04.412098+0000","last_undegraded":"2024-09-16T10:30:04.412098+0000","last_fullsized":"2024-09-16T10:30:04.412098+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:04.076079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076855000000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.314171+0000","last_change":"2024-09-16T10:29:32.784986+0000","last_active":"2024-09-16T10:30:04.314171+0000","last_peered":"2024-09-16T10:30:04.314171+0000","last_clean":"2024-09-16T10:30:04.314171+0000","last_became_active":"2024-09-16T10:29:30.039431+0000","last_became_peered":"2024-09-16T10:29:30.039431+0000","last_unstale":"2024-09-16T10:30:04.314171+0000","last_undegraded":"2024-09-16T10:30:04.314171+0000","last_fullsized":"2024-09-16T10:30:04.314171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:20.175515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041305900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.314672+0000","last_change":"2024-09-16T10:29:32.967651+0000","last_active":"2024-09-16T10:30:04.314672+0000","last_peered":"2024-09-16T10:30:04.314672+0000","last_clean":"2024-09-16T10:30:04.314672+0000","last_became_active":"2024-09-16T10:29:30.043748+0000","last_became_peered":"2024-09-16T10:29:30.043748+0000","last_unstale":"2024-09-16T10:30:04.314672+0000","last_undegraded":"2024-09-16T10:30:04.314672+0000","last_fullsized":"2024-09-16T10:30:04.314672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:37:47.716642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349687,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.411572+0000","last_change":"2024-09-16T10:29:32.794058+0000","last_active":"2024-09-16T10:30:04.411572+0000","last_peered":"2024-09-16T10:30:04.411572+0000","last_clean":"2024-09-16T10:30:04.411572+0000","last_became_active":"2024-09-16T10:29:30.038374+0000","last_became_peered":"2024-09-16T10:29:30.038374+0000","last_unstale":"2024-09-16T10:30:04.411572+0000","last_undegraded":"2024-09-16T10:30:04.411572+0000","last_fullsized":"2024-09-16T10:30:04.411572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:17.735950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040381400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.315003+0000","last_change":"2024-09-16T10:29:29.005269+0000","last_active":"2024-09-16T10:30:04.315003+0000","last_peered":"2024-09-16T10:30:04.315003+0000","last_clean":"2024-09-16T10:30:04.315003+0000","last_became_active":"2024-09-16T10:29:29.004894+0000","last_became_peered":"2024-09-16T10:29:29.004894+0000","last_unstale":"2024-09-16T10:30:04.315003+0000","last_undegraded":"2024-09-16T10:30:04.315003+0000","last_fullsized":"2024-09-16T10:30:04.315003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.994140+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:40.518658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.502,"15min":0.49199999999999999},"min":{"1min":0.39400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66300000000000003,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.47599999999999998,"15min":0.46700000000000003},"min":{"1min":0.40799999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.35599999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.5},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.503,"15min":0.498},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.437}]},{"osd":3,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36599999999999999,"15min":0.36499999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.51000000000000001,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.38800000000000001},"min":{"1min":0.312,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.32000000000000001}]},{"osd":4,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.32300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.38},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.39300000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.308,"15min":0.308},"min":{"1min":0.23200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.39200000000000002,"5min":0.439,"15min":0.439},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.442,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.372}]},{"osd":1,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.34300000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42399999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.34599999999999997,"15min":0.34799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.504,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56200000000000006}]},{"osd":4,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58899999999999997}]},{"osd":5,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.66900000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640287,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.627,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.376,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.50900000000000001,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63700000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.40100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.434}]},{"osd":2,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.40100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.626,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65300000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.55200000000000005,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.33700000000000002}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41899999999999998,"15min":0.41999999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54200000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.438,"15min":0.437},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.42299999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640288,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.505,"15min":0.495},"min":{"1min":0.41099999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68400000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.42299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.48099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68000000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.504},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.46200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.436}]},{"osd":2,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.45300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53500000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45200000000000001}]},{"osd":4,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.32100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.29399999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.48299999999999998}]},{"osd":5,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.52100000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.375,"15min":0.37},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54900000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.378}]},{"osd":2,"last update":"Mon Sep 16 10:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34399999999999997},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.497,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.32200000000000001}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.41099999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.622}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78400000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.57099999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640287,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.32700000000000001},"min":{"1min":0.26800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.44600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.375,"15min":0.379},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.41999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.375}]},{"osd":2,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.26500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.38700000000000001,"15min":0.38900000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.48899999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.31}]},{"osd":3,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.436},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.747,"15min":0.78200000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68899999999999995,"5min":5.032,"15min":5.032},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67600000000000005,"5min":4.9240000000000004,"15min":4.9240000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.75700000000000001,"15min":0.79000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.71999999999999997,"5min":4.9800000000000004,"15min":4.9800000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.753,"5min":5.0030000000000001,"15min":5.0030000000000001},"last":0.48299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:53.178 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:31:53.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.275+0000 7f41f92c9640 10 monclient: get_monmap_and_config 2024-09-16T10:31:53.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.275+0000 7f41f92c9640 10 monclient: build_initial_monmap 2024-09-16T10:31:53.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.275+0000 7f41f92c9640 10 monclient: monmap: 2024-09-16T10:31:53.277 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:53.277 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:53.276178+0000 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:53.276178+0000 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:53.278 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:53.279 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.275+0000 7f41f92c9640 1 Processor -- start 2024-09-16T10:31:53.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 1 -- start start 2024-09-16T10:31:53.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 10 monclient: init 2024-09-16T10:31:53.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:53.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 10 monclient: _add_conns ranks=[3,1,6,0,8,5,4,2,7] 2024-09-16T10:31:53.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 10 monclient(hunting): picked mon.noname-g con 0x7f41f40a38d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:53.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 10 monclient(hunting): picked mon.noname-f con 0x7f41f406c670 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 10 monclient(hunting): picked mon.noname-d con 0x7f41f40a43c0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f4061ae0 con 0x7f41f406c670 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f40a04d0 con 0x7f41f40a38d0 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f40a3250 con 0x7f41f40a43c0 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f92c9640 10 monclient(hunting): authenticate will time out at 880.616578s 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f37fe640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f41f40a43c0 0x7f41f413f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:49078/0 (socket says 172.21.15.94:49078) 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.276+0000 7f41f37fe640 1 -- 172.21.15.94:0/1193104130 learned_addr learned my addr 172.21.15.94:0/1193104130 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.278+0000 7f41f37fe640 1 -- 172.21.15.94:0/1193104130 >> v1:172.21.15.94:6792/0 conn(0x7f41f40a43c0 legacy=0x7f41f413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 -- 172.21.15.94:0/1193104130 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56855180 0 0) 0x7f41f40a3250 con 0x7f41f40a43c0 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:53.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient(hunting): my global_id is 94218 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 -- 172.21.15.94:0/1193104130 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e0003520 con 0x7f41f40a43c0 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f2ffd640 1 -- 172.21.15.94:0/1193104130 >> v1:172.21.15.152:6790/0 conn(0x7f41f40a38d0 legacy=0x7f41f40a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f27fc640 1 -- 172.21.15.94:0/1193104130 >> v1:172.21.15.152:6789/0 conn(0x7f41f406c670 legacy=0x7f41f413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 -- 172.21.15.94:0/1193104130 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986834721 0 0) 0x7f41e0003520 con 0x7f41f40a43c0 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 -- 172.21.15.94:0/1193104130 >> v1:172.21.15.152:6790/0 conn(0x7f41f40a38d0 legacy=0x7f41f40a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 -- 172.21.15.94:0/1193104130 >> v1:172.21.15.152:6789/0 conn(0x7f41f406c670 legacy=0x7f41f413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 monclient: found mon.noname-d 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:31:53.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 -- 172.21.15.94:0/1193104130 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f40a0a00 con 0x7f41f40a43c0 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: _finish_auth 0 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 -- 172.21.15.94:0/1193104130 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f41e4001de0 con 0x7f41f40a43c0 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f92c9640 5 monclient: authenticate success, global_id 94218 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f92c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: dump: 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f92c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:53.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 -- 172.21.15.94:0/1193104130 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41e40020f0 con 0x7f41f40a43c0 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 1 -- 172.21.15.94:0/1193104130 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f41e4005060 con 0x7f41f40a43c0 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f92c9640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f92c9640 4 set_mon_vals no callback set 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f1ffb640 10 monclient: dump: 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.288 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:53.289 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f92c9640 10 monclient: shutdown 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f92c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f92c9640 1 -- 172.21.15.94:0/1193104130 >> v1:172.21.15.94:6792/0 conn(0x7f41f40a43c0 legacy=0x7f41f413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.279+0000 7f41f92c9640 1 -- 172.21.15.94:0/1193104130 shutdown_connections 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.280+0000 7f41f92c9640 1 -- 172.21.15.94:0/1193104130 >> 172.21.15.94:0/1193104130 conn(0x7f41f40a59d0 msgr2=0x7f41f40a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.280+0000 7f41f92c9640 1 -- 172.21.15.94:0/1193104130 shutdown_connections 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.280+0000 7f41f92c9640 1 -- 172.21.15.94:0/1193104130 wait complete. 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.280+0000 7f41f92c9640 10 monclient: build_initial_monmap 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.280+0000 7f41f92c9640 10 monclient: monmap: 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 1 Processor -- start 2024-09-16T10:31:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 1 -- start start 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 10 monclient: init 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 10 monclient: _add_conns ranks=[1,5,4,3,2,8,7,0,6] 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 10 monclient(hunting): picked mon.noname-f con 0x7f41f406c670 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 10 monclient(hunting): picked mon.noname-h con 0x7f41f40a38d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 10 monclient(hunting): picked mon.noname-c con 0x7f41f40a43c0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f4122d20 con 0x7f41f406c670 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f4121110 con 0x7f41f40a43c0 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f4122760 con 0x7f41f40a38d0 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f92c9640 10 monclient(hunting): authenticate will time out at 880.621641s 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f37fe640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f41f40a43c0 0x7f41f41d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:34062/0 (socket says 172.21.15.94:34062) 2024-09-16T10:31:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.281+0000 7f41f37fe640 1 -- 172.21.15.94:0/106652987 learned_addr learned my addr 172.21.15.94:0/106652987 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:53.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.283+0000 7f41f37fe640 1 -- 172.21.15.94:0/106652987 >> v1:172.21.15.94:6791/0 conn(0x7f41f40a43c0 legacy=0x7f41f41d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:53.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.283+0000 7f41f2ffd640 1 -- 172.21.15.94:0/106652987 >> v1:172.21.15.152:6789/0 conn(0x7f41f406c670 legacy=0x7f41f411f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:53.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.283+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683528091 0 0) 0x7f41f4121110 con 0x7f41f40a43c0 2024-09-16T10:31:53.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.284+0000 7f41eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:53.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.284+0000 7f41eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:53.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.284+0000 7f41eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:53.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.284+0000 7f41eb7fe640 10 monclient(hunting): my global_id is 4684 2024-09-16T10:31:53.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.283+0000 7f41f27fc640 1 -- 172.21.15.94:0/106652987 >> v1:172.21.15.152:6791/0 conn(0x7f41f40a38d0 legacy=0x7f41f411fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:53.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.284+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d4003520 con 0x7f41f40a43c0 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.284+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 DispatchQueue::entry inject delay of 0.002000 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702803617 0 0) 0x7f41f4122d20 con 0x7f41f406c670 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): my global_id is 4591 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f4121110 con 0x7f41f406c670 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355084437 0 0) 0x7f41f4122760 con 0x7f41f40a38d0 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): my global_id is 4685 2024-09-16T10:31:53.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f4122d20 con 0x7f41f40a38d0 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849642380 0 0) 0x7f41d4003520 con 0x7f41f40a43c0 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41f4122760 con 0x7f41f40a43c0 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f41e4004ea0 con 0x7f41f40a43c0 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): dump: 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2020847701 0 0) 0x7f41f4122760 con 0x7f41f40a43c0 2024-09-16T10:31:53.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 >> v1:172.21.15.152:6791/0 conn(0x7f41f40a38d0 legacy=0x7f41f411fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 >> v1:172.21.15.152:6789/0 conn(0x7f41f406c670 legacy=0x7f41f411f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 monclient: found mon.c 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f4120340 con 0x7f41f40a43c0 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41f92c9640 5 monclient: authenticate success, global_id 4684 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41f92c9640 10 monclient: _renew_subs 2024-09-16T10:31:53.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41f92c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f4120950 con 0x7f41f40a43c0 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41f92c9640 10 monclient: _renew_subs 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41f92c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f409ec10 con 0x7f41f40a43c0 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41e40050c0 con 0x7f41f40a43c0 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f41e4005880 con 0x7f41f40a43c0 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41eb7fe640 10 monclient: dump: 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.286+0000 7f41f17fa640 4 set_mon_vals no callback set 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.287+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f41e40108e0 con 0x7f41f40a43c0 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.287+0000 7f41f92c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.287+0000 7f41f92c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.287+0000 7f41f92c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.287+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f4121520 con 0x7f41f40a43c0 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.287+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f41e4046360 con 0x7f41f40a43c0 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.287+0000 7f41f2ffd640 10 monclient: get_auth_request con 0x7f41d4039250 auth_method 0 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.291+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f41e40158e0 con 0x7f41f40a43c0 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.292+0000 7f41eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:53.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.292+0000 7f41eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:53.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.454+0000 7f41f92c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:31:53.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.454+0000 7f41f92c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:31:53.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.454+0000 7f41f92c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:53.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.454+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 --> v1:172.21.15.94:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41f406b6a0 con 0x7f41f40a43c0 2024-09-16T10:31:53.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.455+0000 7f41eb7fe640 1 -- 172.21.15.94:0/106652987 <== mon.4 v1:172.21.15.94:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+10746 (unknown 861009196 0 1988707045) 0x7f41e40154d0 con 0x7f41f40a43c0 2024-09-16T10:31:53.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.455+0000 7f41eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:31:53.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.455+0000 7f41eb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:31:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:31:53.457 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":22,"fsid":"f51dbe5c-da09-4bb3-8cd3-867d68869245","created":"2024-09-16T10:29:16.029056+0000","modified":"2024-09-16T10:30:04.309090+0000","last_up_change":"2024-09-16T10:29:27.994140+0000","last_in_change":"2024-09-16T10:29:20.181320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:29:27.046989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T10:29:28.942998+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"19cab4f7-c271-4ce8-895e-75a5871502f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":4202994153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":4202994153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":4202994153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":4202994153}]},"public_addr":"172.21.15.94:6804/4202994153","cluster_addr":"172.21.15.94:6806/4202994153","heartbeat_back_addr":"172.21.15.94:6810/4202994153","heartbeat_front_addr":"172.21.15.94:6808/4202994153","state":["exists","up"]},{"osd":1,"uuid":"6641fac5-e871-4401-bf30-598b633e70b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":782646670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":782646670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":782646670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":782646670}]},"public_addr":"172.21.15.94:6800/782646670","cluster_addr":"172.21.15.94:6801/782646670","heartbeat_back_addr":"172.21.15.94:6803/782646670","heartbeat_front_addr":"172.21.15.94:6802/782646670","state":["exists","up"]},{"osd":2,"uuid":"6f4cedf2-071d-44f9-87a3-a766668fcd03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3102113414}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3102113414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3102113414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3102113414}]},"public_addr":"172.21.15.94:6805/3102113414","cluster_addr":"172.21.15.94:6807/3102113414","heartbeat_back_addr":"172.21.15.94:6811/3102113414","heartbeat_front_addr":"172.21.15.94:6809/3102113414","state":["exists","up"]},{"osd":3,"uuid":"3634242e-61e0-4221-8c38-7dce4d5053ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6805","nonce":823649059}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6807","nonce":823649059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6811","nonce":823649059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6809","nonce":823649059}]},"public_addr":"172.21.15.152:6805/823649059","cluster_addr":"172.21.15.152:6807/823649059","heartbeat_back_addr":"172.21.15.152:6811/823649059","heartbeat_front_addr":"172.21.15.152:6809/823649059","state":["exists","up"]},{"osd":4,"uuid":"eed750dc-2d29-443f-a91c-d9c5abdcf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6801","nonce":1475699231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6802","nonce":1475699231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6804","nonce":1475699231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6803","nonce":1475699231}]},"public_addr":"172.21.15.152:6801/1475699231","cluster_addr":"172.21.15.152:6802/1475699231","heartbeat_back_addr":"172.21.15.152:6804/1475699231","heartbeat_front_addr":"172.21.15.152:6803/1475699231","state":["exists","up"]},{"osd":5,"uuid":"5d6daac7-1ac7-4f0a-b315-27d1333fc3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6806","nonce":3657457384}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6808","nonce":3657457384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6812","nonce":3657457384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.152:6810","nonce":3657457384}]},"public_addr":"172.21.15.152:6806/3657457384","cluster_addr":"172.21.15.152:6808/3657457384","heartbeat_back_addr":"172.21.15.152:6812/3657457384","heartbeat_front_addr":"172.21.15.152:6810/3657457384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.818679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.874108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.957469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.872051+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.780789+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:29:25.901652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:31:53.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.457+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 >> v1:172.21.15.152:6800/2686360781 conn(0x7f41d4039250 legacy=0x7f41d403b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.457+0000 7f41f92c9640 10 monclient: shutdown 2024-09-16T10:31:53.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.457+0000 7f41f92c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:53.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.457+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 >> v1:172.21.15.94:6791/0 conn(0x7f41f40a43c0 legacy=0x7f41f41d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:53.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.457+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 shutdown_connections 2024-09-16T10:31:53.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.457+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 >> 172.21.15.94:0/106652987 conn(0x7f41f40a59d0 msgr2=0x7f41f409f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:53.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.457+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 shutdown_connections 2024-09-16T10:31:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:53.457+0000 7f41f92c9640 1 -- 172.21.15.94:0/106652987 wait complete. 2024-09-16T10:31:54.470 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T10:31:54.471 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T10:31:54.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.580+0000 7f25ec501640 10 monclient: get_monmap_and_config 2024-09-16T10:31:54.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.580+0000 7f25ec501640 10 monclient: build_initial_monmap 2024-09-16T10:31:54.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.580+0000 7f25ec501640 10 monclient: monmap: 2024-09-16T10:31:54.582 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:54.582 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:54.581636+0000 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:54.581636+0000 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:54.583 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.581+0000 7f25ec501640 1 Processor -- start 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.581+0000 7f25ec501640 1 -- start start 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.581+0000 7f25ec501640 10 monclient: init 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 10 monclient: _add_conns ranks=[5,4,0,6,1,8,3,2,7] 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 10 monclient(hunting): picked mon.noname-h con 0x7f25e409ba70 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 10 monclient(hunting): picked mon.noname-c con 0x7f25e409e860 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 10 monclient(hunting): picked mon.noname-a con 0x7f25e413aab0 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e4061ae0 con 0x7f25e413aab0 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e409be50 con 0x7f25e409e860 2024-09-16T10:31:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e40a1530 con 0x7f25e409ba70 2024-09-16T10:31:54.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:54.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25ec501640 10 monclient(hunting): authenticate will time out at 881.922303s 2024-09-16T10:31:54.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25e9a9c640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f25e409e860 0x7f25e4139390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:34078/0 (socket says 172.21.15.94:34078) 2024-09-16T10:31:54.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25eaa9e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f25e413aab0 0x7f25e413cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53482/0 (socket says 172.21.15.94:53482) 2024-09-16T10:31:54.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.582+0000 7f25e9a9c640 1 -- 172.21.15.94:0/1928399467 learned_addr learned my addr 172.21.15.94:0/1928399467 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:54.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e9a9c640 1 -- 172.21.15.94:0/1928399467 >> v1:172.21.15.94:6791/0 conn(0x7f25e409e860 legacy=0x7f25e4139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:54.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25eaa9e640 1 -- 172.21.15.94:0/1928399467 >> v1:172.21.15.94:6789/0 conn(0x7f25e413aab0 legacy=0x7f25e413cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:54.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 1 -- 172.21.15.94:0/1928399467 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459759049 0 0) 0x7f25e4061ae0 con 0x7f25e413aab0 2024-09-16T10:31:54.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient(hunting): _negotiate 2024-09-16T10:31:54.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:54.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient(hunting): my global_id is 4590 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 1 -- 172.21.15.94:0/1928399467 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d0003520 con 0x7f25e413aab0 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25ea29d640 1 -- 172.21.15.94:0/1928399467 >> v1:172.21.15.152:6791/0 conn(0x7f25e409ba70 legacy=0x7f25e40714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 1 -- 172.21.15.94:0/1928399467 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1935986056 0 0) 0x7f25d0003520 con 0x7f25e413aab0 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 1 -- 172.21.15.94:0/1928399467 >> v1:172.21.15.152:6791/0 conn(0x7f25e409ba70 legacy=0x7f25e40714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 1 -- 172.21.15.94:0/1928399467 >> v1:172.21.15.94:6791/0 conn(0x7f25e409e860 legacy=0x7f25e4139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient: _finish_hunting 0 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 1 monclient: found mon.noname-a 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:31:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 1 -- 172.21.15.94:0/1928399467 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e40a0c20 con 0x7f25e413aab0 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient: _finish_auth 0 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient: _check_auth_tickets 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 1 -- 172.21.15.94:0/1928399467 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25dc001d10 con 0x7f25e413aab0 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25ec501640 5 monclient: authenticate success, global_id 4590 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25ec501640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.584+0000 7f25e929b640 10 monclient: dump: 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:54.591 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:54.592 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25e929b640 1 -- 172.21.15.94:0/1928399467 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25dc002020 con 0x7f25e413aab0 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25e929b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25e929b640 1 -- 172.21.15.94:0/1928399467 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25dc004ea0 con 0x7f25e413aab0 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 4 set_mon_vals no callback set 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25e929b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25e929b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25e929b640 10 monclient: dump: 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:54.593 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 10 monclient: shutdown 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 1 -- 172.21.15.94:0/1928399467 >> v1:172.21.15.94:6789/0 conn(0x7f25e413aab0 legacy=0x7f25e413cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 1 -- 172.21.15.94:0/1928399467 shutdown_connections 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 1 -- 172.21.15.94:0/1928399467 >> 172.21.15.94:0/1928399467 conn(0x7f25e40a1780 msgr2=0x7f25e40a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 1 -- 172.21.15.94:0/1928399467 shutdown_connections 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.585+0000 7f25ec501640 1 -- 172.21.15.94:0/1928399467 wait complete. 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.586+0000 7f25ec501640 10 monclient: build_initial_monmap 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.586+0000 7f25ec501640 10 monclient: monmap: 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.586+0000 7f25ec501640 1 Processor -- start 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.586+0000 7f25ec501640 1 -- start start 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.586+0000 7f25ec501640 10 monclient: init 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:54.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 10 monclient: _add_conns ranks=[4,2,5,3,1,7,8,0,6] 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 10 monclient(hunting): picked mon.noname-c con 0x7f25e409ba70 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 10 monclient(hunting): picked mon.noname-b con 0x7f25e409e860 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 10 monclient(hunting): picked mon.noname-h con 0x7f25e413aab0 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e406f3f0 con 0x7f25e409e860 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e4072400 con 0x7f25e409ba70 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e413a340 con 0x7f25e413aab0 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ec501640 10 monclient(hunting): authenticate will time out at 881.927125s 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ea29d640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f25e409ba70 0x7f25e4070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:34094/0 (socket says 172.21.15.94:34094) 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25ea29d640 1 -- 172.21.15.94:0/3780421966 learned_addr learned my addr 172.21.15.94:0/3780421966 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.587+0000 7f25e9a9c640 1 --1- 172.21.15.94:0/3780421966 >> v1:172.21.15.94:6790/0 conn(0x7f25e409e860 0x7f25e41d0360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41120/0 (socket says 172.21.15.94:41120) 2024-09-16T10:31:54.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.589+0000 7f25e9a9c640 1 -- 172.21.15.94:0/3780421966 >> v1:172.21.15.94:6790/0 conn(0x7f25e409e860 legacy=0x7f25e41d0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:54.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.589+0000 7f25ea29d640 1 -- 172.21.15.94:0/3780421966 >> v1:172.21.15.94:6791/0 conn(0x7f25e409ba70 legacy=0x7f25e4070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:54.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.589+0000 7f25eaa9e640 1 -- 172.21.15.94:0/3780421966 >> v1:172.21.15.152:6791/0 conn(0x7f25e413aab0 legacy=0x7f25e41d3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:54.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927403206 0 0) 0x7f25e413a340 con 0x7f25e413aab0 2024-09-16T10:31:54.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:54.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:54.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:54.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): my global_id is 4694 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c8003520 con 0x7f25e413aab0 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25ea29d640 1 --1- 172.21.15.94:0/3780421966 >> v1:172.21.15.94:6791/0 conn(0x7f25e409ba70 0x7f25e4070920 :-1 s=OPENED pgs=170 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7f25e4072400 auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801466910 0 0) 0x7f25e406f3f0 con 0x7f25e409e860 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): my global_id is 4619 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e413a340 con 0x7f25e409e860 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1220630913 0 0) 0x7f25c8003520 con 0x7f25e413aab0 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25e406f3f0 con 0x7f25e413aab0 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25dc0015d0 con 0x7f25e413aab0 2024-09-16T10:31:54.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): dump: 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2391001319 0 0) 0x7f25e413a340 con 0x7f25e409e860 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25c8003520 con 0x7f25e409e860 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25d4001c10 con 0x7f25e409e860 2024-09-16T10:31:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient(hunting): dump: 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 397697318 0 0) 0x7f25e406f3f0 con 0x7f25e413aab0 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 >> v1:172.21.15.94:6791/0 conn(0x7f25e409ba70 legacy=0x7f25e4070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 >> v1:172.21.15.94:6790/0 conn(0x7f25e409e860 legacy=0x7f25e41d0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 monclient: found mon.h 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:31:55.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.152:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e40717f0 con 0x7f25e413aab0 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25ec501640 5 monclient: authenticate success, global_id 4694 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960749731 0 0) 0x7f25e4072400 con 0x7f25e409ba70 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25daffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25ec501640 10 monclient: _renew_subs 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25ec501640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.994+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.152:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e409c880 con 0x7f25e413aab0 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25ec501640 10 monclient: _renew_subs 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25ec501640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.152:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e409daa0 con 0x7f25e413aab0 2024-09-16T10:31:55.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.5 v1:172.21.15.152:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25dc005150 con 0x7f25e413aab0 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.5 v1:172.21.15.152:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f25dc001cf0 con 0x7f25e413aab0 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25daffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25e8a9a640 4 set_mon_vals no callback set 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25daffd640 10 monclient: dump: 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25ec501640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.152:6791/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f25cc000f10 con 0x7f25e413aab0 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.995+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.5 v1:172.21.15.152:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f25dc005680 con 0x7f25e413aab0 2024-09-16T10:31:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.996+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.5 v1:172.21.15.152:6791/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f25dc046180 con 0x7f25e413aab0 2024-09-16T10:31:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.996+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.94:6804/4202994153 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25c8043e00 con 0x7f25c8040190 2024-09-16T10:31:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.996+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== mon.5 v1:172.21.15.152:6791/0 9 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (unknown 3265615600 0 0) 0x7f25dc046600 con 0x7f25e413aab0 2024-09-16T10:31:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.996+0000 7f25daffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2024-09-16T10:31:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.996+0000 7f25ea29d640 10 monclient: get_auth_request con 0x7f25c80397b0 auth_method 0 2024-09-16T10:31:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:54.996+0000 7f25e9a9c640 10 monclient: get_auth_request con 0x7f25c8040190 auth_method 0 2024-09-16T10:31:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.001+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== osd.0 v1:172.21.15.94:6804/4202994153 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f25c8043e00 con 0x7f25c8040190 2024-09-16T10:31:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.013+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 --> v1:172.21.15.94:6804/4202994153 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f25cc002b50 con 0x7f25c8040190 2024-09-16T10:31:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.014+0000 7f25daffd640 1 -- 172.21.15.94:0/3780421966 <== osd.0 v1:172.21.15.94:6804/4202994153 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f25cc002b50 con 0x7f25c8040190 2024-09-16T10:31:55.016 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:55.016 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T10:31:55.016 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.014+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 >> v1:172.21.15.94:6804/4202994153 conn(0x7f25c8040190 legacy=0x7f25c80425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.014+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 >> v1:172.21.15.152:6800/2686360781 conn(0x7f25c80397b0 legacy=0x7f25c803bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.014+0000 7f25ec501640 10 monclient: shutdown 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.014+0000 7f25ec501640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.014+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 >> v1:172.21.15.152:6791/0 conn(0x7f25e413aab0 legacy=0x7f25e41d3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.014+0000 7f25eaa9e640 1 -- 172.21.15.94:0/3780421966 reap_dead start 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.015+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 shutdown_connections 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.015+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 >> 172.21.15.94:0/3780421966 conn(0x7f25e40a1780 msgr2=0x7f25e40a2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.015+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 shutdown_connections 2024-09-16T10:31:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.015+0000 7f25ec501640 1 -- 172.21.15.94:0/3780421966 wait complete. 2024-09-16T10:31:55.028 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T10:31:55.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.126+0000 7efdd54b1640 10 monclient: get_monmap_and_config 2024-09-16T10:31:55.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.126+0000 7efdd54b1640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.126+0000 7efdd54b1640 10 monclient: monmap: 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:55.126819+0000 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:55.126819+0000 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.128 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.129 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.129 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.126+0000 7efdd54b1640 1 Processor -- start 2024-09-16T10:31:55.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.126+0000 7efdd54b1640 1 -- start start 2024-09-16T10:31:55.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.126+0000 7efdd54b1640 10 monclient: init 2024-09-16T10:31:55.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 10 monclient: _add_conns ranks=[8,3,0,5,1,6,4,7,2] 2024-09-16T10:31:55.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 10 monclient(hunting): picked mon.noname-e con 0x7efdd00a3860 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:55.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 10 monclient(hunting): picked mon.noname-g con 0x7efdd006c1d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:55.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 10 monclient(hunting): picked mon.noname-a con 0x7efdd00a4350 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:55.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd0061ae0 con 0x7efdd00a4350 2024-09-16T10:31:55.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00a0600 con 0x7efdd006c1d0 2024-09-16T10:31:55.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00a0b30 con 0x7efdd00a3860 2024-09-16T10:31:55.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdd54b1640 10 monclient(hunting): authenticate will time out at 882.467278s 2024-09-16T10:31:55.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdcf7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efdd00a4350 0x7efdd013ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53484/0 (socket says 172.21.15.94:53484) 2024-09-16T10:31:55.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdcf7fe640 1 -- 172.21.15.94:0/3461036405 learned_addr learned my addr 172.21.15.94:0/3461036405 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.127+0000 7efdceffd640 1 --1- 172.21.15.94:0/3461036405 >> v1:172.21.15.94:6793/0 conn(0x7efdd00a3860 0x7efdd00a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:44096/0 (socket says 172.21.15.94:44096) 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcf7fe640 1 -- 172.21.15.94:0/3461036405 >> v1:172.21.15.94:6789/0 conn(0x7efdd00a4350 legacy=0x7efdd013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdceffd640 1 -- 172.21.15.94:0/3461036405 >> v1:172.21.15.94:6793/0 conn(0x7efdd00a3860 legacy=0x7efdd00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446240208 0 0) 0x7efdd00a0b30 con 0x7efdd00a3860 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient(hunting): my global_id is 4652 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb8003520 con 0x7efdd00a3860 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543188828 0 0) 0x7efdd0061ae0 con 0x7efdd00a4350 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient(hunting): my global_id is 4599 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd00a0b30 con 0x7efdd00a4350 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdce7fc640 1 -- 172.21.15.94:0/3461036405 >> v1:172.21.15.152:6790/0 conn(0x7efdd006c1d0 legacy=0x7efdd013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90238760 0 0) 0x7efdb8003520 con 0x7efdd00a3860 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 >> v1:172.21.15.152:6790/0 conn(0x7efdd006c1d0 legacy=0x7efdd013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 >> v1:172.21.15.94:6789/0 conn(0x7efdd00a4350 legacy=0x7efdd013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 monclient: found mon.noname-e 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.94:6793/0 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 --> v1:172.21.15.94:6793/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd00a0e10 con 0x7efdd00a3860 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdd54b1640 5 monclient: authenticate success, global_id 4652 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efdc0001990 con 0x7efdd00a3860 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdd54b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdcdffb640 10 monclient: dump: 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.137 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.129+0000 7efdd54b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 <== mon.8 v1:172.21.15.94:6793/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc0001ca0 con 0x7efdd00a3860 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdcdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdcdffb640 1 -- 172.21.15.94:0/3461036405 <== mon.8 v1:172.21.15.94:6793/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efdc0004b20 con 0x7efdd00a3860 2024-09-16T10:31:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdd54b1640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdd54b1640 4 set_mon_vals no callback set 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdcdffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdcdffb640 10 monclient: dump: 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdd54b1640 10 monclient: shutdown 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdd54b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdd54b1640 1 -- 172.21.15.94:0/3461036405 >> v1:172.21.15.94:6793/0 conn(0x7efdd00a3860 legacy=0x7efdd00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdd54b1640 1 -- 172.21.15.94:0/3461036405 shutdown_connections 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdd54b1640 1 -- 172.21.15.94:0/3461036405 >> 172.21.15.94:0/3461036405 conn(0x7efdd00a5960 msgr2=0x7efdd00a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdd54b1640 1 -- 172.21.15.94:0/3461036405 shutdown_connections 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.130+0000 7efdd54b1640 1 -- 172.21.15.94:0/3461036405 wait complete. 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.131+0000 7efdd54b1640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.131+0000 7efdd54b1640 10 monclient: monmap: 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.131+0000 7efdd54b1640 1 Processor -- start 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.131+0000 7efdd54b1640 1 -- start start 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.131+0000 7efdd54b1640 10 monclient: init 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 10 monclient: _add_conns ranks=[7,0,4,8,6,2,1,5,3] 2024-09-16T10:31:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 10 monclient(hunting): picked mon.noname-i con 0x7efdd006c1d0 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 10 monclient(hunting): picked mon.noname-a con 0x7efdd00a3860 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 10 monclient(hunting): picked mon.noname-c con 0x7efdd00a4350 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:55.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd01abf40 con 0x7efdd00a3860 2024-09-16T10:31:55.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd006b980 con 0x7efdd00a4350 2024-09-16T10:31:55.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd009efb0 con 0x7efdd006c1d0 2024-09-16T10:31:55.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdd54b1640 10 monclient(hunting): authenticate will time out at 882.472202s 2024-09-16T10:31:55.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdce7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efdd00a3860 0x7efdd01cfb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53496/0 (socket says 172.21.15.94:53496) 2024-09-16T10:31:55.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.132+0000 7efdce7fc640 1 -- 172.21.15.94:0/2607809772 learned_addr learned my addr 172.21.15.94:0/2607809772 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdce7fc640 1 -- 172.21.15.94:0/2607809772 >> v1:172.21.15.94:6789/0 conn(0x7efdd00a3860 legacy=0x7efdd01cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdcf7fe640 1 -- 172.21.15.94:0/2607809772 >> v1:172.21.15.94:6791/0 conn(0x7efdd00a4350 legacy=0x7efdd01d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdceffd640 1 -- 172.21.15.94:0/2607809772 >> v1:172.21.15.152:6792/0 conn(0x7efdd006c1d0 legacy=0x7efdd01be560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158928415 0 0) 0x7efdd01abf40 con 0x7efdd00a3860 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): my global_id is 4608 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdbc003550 con 0x7efdd00a3860 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45177176 0 0) 0x7efdd006b980 con 0x7efdd00a4350 2024-09-16T10:31:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): my global_id is 4702 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd01abf40 con 0x7efdd00a4350 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49727053 0 0) 0x7efdd009efb0 con 0x7efdd006c1d0 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): my global_id is 4705 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd006b980 con 0x7efdd006c1d0 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2826636613 0 0) 0x7efdbc003550 con 0x7efdd00a3860 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdd009efb0 con 0x7efdd00a3860 2024-09-16T10:31:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efdc4003610 con 0x7efdd00a3860 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): dump: 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1619957247 0 0) 0x7efdd01abf40 con 0x7efdd00a4350 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdbc003550 con 0x7efdd00a4350 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efdc8001fe0 con 0x7efdd00a4350 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient(hunting): dump: 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.152 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1358337748 0 0) 0x7efdd009efb0 con 0x7efdd00a3860 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 >> v1:172.21.15.152:6792/0 conn(0x7efdd006c1d0 legacy=0x7efdd01be560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 >> v1:172.21.15.94:6791/0 conn(0x7efdd00a4350 legacy=0x7efdd01d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 1 monclient: found mon.a 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.134+0000 7efdb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd009f480 con 0x7efdd00a3860 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 5 monclient: authenticate success, global_id 4608 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 10 monclient: _renew_subs 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd01aa530 con 0x7efdd00a3860 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 10 monclient: _renew_subs 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd01bee50 con 0x7efdd00a3860 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc4003b00 con 0x7efdd00a3860 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efdc4003fb0 con 0x7efdd00a3860 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdcd7fa640 4 set_mon_vals no callback set 2024-09-16T10:31:55.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 10 monclient: dump: 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.155 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7efdc4004110 con 0x7efdd00a3860 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.135+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb0005040 con 0x7efdd00a3860 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.136+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7efdc4045ef0 con 0x7efdd00a3860 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.136+0000 7efdceffd640 10 monclient: get_auth_request con 0x7efdbc039aa0 auth_method 0 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.140+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7efdc40153a0 con 0x7efdd00a3860 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.140+0000 7efdb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:55.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.140+0000 7efdb77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:55.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.301+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7efdb0000b90 con 0x7efdbc039aa0 2024-09-16T10:31:55.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.302+0000 7efdb77fe640 1 -- 172.21.15.94:0/2607809772 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (unknown 2057319476 0 0) 0x7efdb0000b90 con 0x7efdbc039aa0 2024-09-16T10:31:55.304 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T10:31:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.303+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 >> v1:172.21.15.152:6800/2686360781 conn(0x7efdbc039aa0 legacy=0x7efdbc03bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.303+0000 7efdd54b1640 10 monclient: shutdown 2024-09-16T10:31:55.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.303+0000 7efdd54b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.303+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 >> v1:172.21.15.94:6789/0 conn(0x7efdd00a3860 legacy=0x7efdd01cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.303+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 shutdown_connections 2024-09-16T10:31:55.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.303+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 >> 172.21.15.94:0/2607809772 conn(0x7efdd00a5960 msgr2=0x7efdd009ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.304+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 shutdown_connections 2024-09-16T10:31:55.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.304+0000 7efdd54b1640 1 -- 172.21.15.94:0/2607809772 wait complete. 2024-09-16T10:31:55.315 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T10:31:55.316 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T10:31:55.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.413+0000 7f819acc9640 10 monclient: get_monmap_and_config 2024-09-16T10:31:55.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.413+0000 7f819acc9640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.413+0000 7f819acc9640 10 monclient: monmap: 2024-09-16T10:31:55.415 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.415 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.415 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:55.414117+0000 2024-09-16T10:31:55.415 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:55.414117+0000 2024-09-16T10:31:55.415 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.415 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.415 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.416 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.416 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.416 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.416 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.416 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.416 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.416 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.416 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.416 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 1 Processor -- start 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 1 -- start start 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 10 monclient: init 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 10 monclient: _add_conns ranks=[2,6,8,7,1,3,4,0,5] 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 10 monclient(hunting): picked mon.noname-b con 0x7f81940a20d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 10 monclient(hunting): picked mon.noname-d con 0x7f81940a1600 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 10 monclient(hunting): picked mon.noname-e con 0x7f81940a19e0 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8194061ae0 con 0x7f81940a20d0 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819409ff70 con 0x7f81940a1600 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81940a0640 con 0x7f81940a19e0 2024-09-16T10:31:55.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f819acc9640 10 monclient(hunting): authenticate will time out at 882.754700s 2024-09-16T10:31:55.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f8193fff640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f81940a1600 0x7f819413b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:49086/0 (socket says 172.21.15.94:49086) 2024-09-16T10:31:55.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f8199266640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f81940a19e0 0x7f819413ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:44112/0 (socket says 172.21.15.94:44112) 2024-09-16T10:31:55.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.414+0000 7f8193fff640 1 -- 172.21.15.94:0/2130472048 learned_addr learned my addr 172.21.15.94:0/2130472048 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.416+0000 7f8199266640 1 -- 172.21.15.94:0/2130472048 >> v1:172.21.15.94:6793/0 conn(0x7f81940a19e0 legacy=0x7f819413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.416+0000 7f8198a65640 1 -- 172.21.15.94:0/2130472048 >> v1:172.21.15.94:6790/0 conn(0x7f81940a20d0 legacy=0x7f81940a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f8193fff640 1 -- 172.21.15.94:0/2130472048 >> v1:172.21.15.94:6792/0 conn(0x7f81940a1600 legacy=0x7f819413b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611086222 0 0) 0x7f8194061ae0 con 0x7f81940a20d0 2024-09-16T10:31:55.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): my global_id is 4628 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8184003520 con 0x7f81940a20d0 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93054300 0 0) 0x7f81940a0640 con 0x7f81940a19e0 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): my global_id is 4661 2024-09-16T10:31:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8194061ae0 con 0x7f81940a19e0 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97923598 0 0) 0x7f819409ff70 con 0x7f81940a1600 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient(hunting): my global_id is 94227 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81940a0640 con 0x7f81940a1600 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 359600455 0 0) 0x7f81940a0640 con 0x7f81940a1600 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f8198a65640 1 --1- 172.21.15.94:0/2130472048 >> v1:172.21.15.94:6790/0 conn(0x7f81940a20d0 0x7f81940a24b0 :-1 s=OPENED pgs=163 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7f8184003520 auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 >> v1:172.21.15.94:6793/0 conn(0x7f81940a19e0 legacy=0x7f819413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 >> v1:172.21.15.94:6790/0 conn(0x7f81940a20d0 legacy=0x7f81940a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 monclient: found mon.noname-d 2024-09-16T10:31:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:31:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81940a3000 con 0x7f81940a1600 2024-09-16T10:31:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8180001980 con 0x7f81940a1600 2024-09-16T10:31:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f819acc9640 5 monclient: authenticate success, global_id 94227 2024-09-16T10:31:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f819acc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: dump: 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.428 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.429 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.429 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.429 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.429 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.429 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.429 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.429 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f81880019e0 con 0x7f81940a20d0 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f819acc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8180001c90 con 0x7f81940a1600 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 1 -- 172.21.15.94:0/2130472048 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8180004be0 con 0x7f81940a1600 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f819acc9640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f819acc9640 4 set_mon_vals no callback set 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.417+0000 7f81937fe640 10 monclient: dump: 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.430 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.418+0000 7f819acc9640 10 monclient: shutdown 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.418+0000 7f819acc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.418+0000 7f819acc9640 1 -- 172.21.15.94:0/2130472048 >> v1:172.21.15.94:6792/0 conn(0x7f81940a1600 legacy=0x7f819413b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.418+0000 7f819acc9640 1 -- 172.21.15.94:0/2130472048 shutdown_connections 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.418+0000 7f819acc9640 1 -- 172.21.15.94:0/2130472048 >> 172.21.15.94:0/2130472048 conn(0x7f81940a5830 msgr2=0x7f81940a5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.418+0000 7f819acc9640 1 -- 172.21.15.94:0/2130472048 shutdown_connections 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.418+0000 7f819acc9640 1 -- 172.21.15.94:0/2130472048 wait complete. 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.418+0000 7f819acc9640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.418+0000 7f819acc9640 10 monclient: monmap: 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.433 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 1 Processor -- start 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 1 -- start start 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 10 monclient: init 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 10 monclient: _add_conns ranks=[5,2,1,6,8,3,4,0,7] 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 10 monclient(hunting): picked mon.noname-h con 0x7f81940a1600 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 10 monclient(hunting): picked mon.noname-b con 0x7f81940a19e0 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 10 monclient(hunting): picked mon.noname-f con 0x7f81940a20d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819409f330 con 0x7f81940a20d0 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8194127050 con 0x7f81940a19e0 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8194125b80 con 0x7f81940a1600 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f819acc9640 10 monclient(hunting): authenticate will time out at 882.759695s 2024-09-16T10:31:55.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f8193fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f81940a19e0 0x7f81941cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:46144/0 (socket says 172.21.15.94:46144) 2024-09-16T10:31:55.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.419+0000 7f8193fff640 1 -- 172.21.15.94:0/1859694776 learned_addr learned my addr 172.21.15.94:0/1859694776 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.421+0000 7f8193fff640 1 -- 172.21.15.94:0/1859694776 >> v1:172.21.15.94:6790/0 conn(0x7f81940a19e0 legacy=0x7f81941cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.421+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261626935 0 0) 0x7f8194127050 con 0x7f81940a19e0 2024-09-16T10:31:55.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.421+0000 7f81917fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.421+0000 7f81917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.421+0000 7f81917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.421+0000 7f81917fa640 10 monclient(hunting): my global_id is 4637 2024-09-16T10:31:55.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8178003520 con 0x7f81940a19e0 2024-09-16T10:31:55.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f8199266640 1 -- 172.21.15.94:0/1859694776 >> v1:172.21.15.152:6789/0 conn(0x7f81940a20d0 legacy=0x7f81941d2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f8198a65640 1 -- 172.21.15.94:0/1859694776 >> v1:172.21.15.152:6791/0 conn(0x7f81940a1600 legacy=0x7f819413f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2953917527 0 0) 0x7f8178003520 con 0x7f81940a19e0 2024-09-16T10:31:55.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8194127050 con 0x7f81940a19e0 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8180004b20 con 0x7f81940a19e0 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): dump: 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618112128 0 0) 0x7f819409f330 con 0x7f81940a20d0 2024-09-16T10:31:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): my global_id is 4600 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8178003520 con 0x7f81940a20d0 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090359020 0 0) 0x7f8194125b80 con 0x7f81940a1600 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient(hunting): my global_id is 4703 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f819409f330 con 0x7f81940a1600 2024-09-16T10:31:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 482718180 0 0) 0x7f8194127050 con 0x7f81940a19e0 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 >> v1:172.21.15.152:6791/0 conn(0x7f81940a1600 legacy=0x7f819413f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 >> v1:172.21.15.152:6789/0 conn(0x7f81940a20d0 legacy=0x7f81941d2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 monclient: found mon.b 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819409ede0 con 0x7f81940a19e0 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f819acc9640 5 monclient: authenticate success, global_id 4637 2024-09-16T10:31:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f819acc9640 10 monclient: _renew_subs 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f819acc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81940a7560 con 0x7f81940a19e0 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f819acc9640 10 monclient: _renew_subs 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f819acc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81941267b0 con 0x7f81940a19e0 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8180001320 con 0x7f81940a19e0 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f8180005550 con 0x7f81940a19e0 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f8192ffd640 4 set_mon_vals no callback set 2024-09-16T10:31:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.422+0000 7f81917fa640 10 monclient: dump: 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.423+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f81800105b0 con 0x7f81940a19e0 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.423+0000 7f819acc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.423+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.94:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8174000f10 con 0x7f81940a19e0 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.423+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f81800460d0 con 0x7f81940a19e0 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.423+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.94:6800/782646670 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8178043a60 con 0x7f817803fdf0 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.423+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (unknown 3265615600 0 0) 0x7f8180046550 con 0x7f81940a19e0 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.423+0000 7f81917fa640 10 monclient: handle_get_version_reply finishing 1 version 22 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.423+0000 7f8198a65640 10 monclient: get_auth_request con 0x7f8178039220 auth_method 0 2024-09-16T10:31:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.423+0000 7f8199266640 10 monclient: get_auth_request con 0x7f817803fdf0 auth_method 0 2024-09-16T10:31:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.428+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== osd.1 v1:172.21.15.94:6800/782646670 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f8178043a60 con 0x7f817803fdf0 2024-09-16T10:31:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.440+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 --> v1:172.21.15.94:6800/782646670 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8174002b50 con 0x7f817803fdf0 2024-09-16T10:31:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.440+0000 7f81917fa640 1 -- 172.21.15.94:0/1859694776 <== osd.1 v1:172.21.15.94:6800/782646670 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f8174002b50 con 0x7f817803fdf0 2024-09-16T10:31:55.447 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T10:31:55.448 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.440+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 >> v1:172.21.15.94:6800/782646670 conn(0x7f817803fdf0 legacy=0x7f8178042210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.440+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 >> v1:172.21.15.152:6800/2686360781 conn(0x7f8178039220 legacy=0x7f817803b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.440+0000 7f819acc9640 10 monclient: shutdown 2024-09-16T10:31:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.440+0000 7f819acc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.440+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 >> v1:172.21.15.94:6790/0 conn(0x7f81940a19e0 legacy=0x7f81941cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.440+0000 7f8199266640 1 -- 172.21.15.94:0/1859694776 reap_dead start 2024-09-16T10:31:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.441+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 shutdown_connections 2024-09-16T10:31:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.441+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 >> 172.21.15.94:0/1859694776 conn(0x7f81940a5830 msgr2=0x7f81940a7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.441+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 shutdown_connections 2024-09-16T10:31:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.441+0000 7f819acc9640 1 -- 172.21.15.94:0/1859694776 wait complete. 2024-09-16T10:31:55.452 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T10:31:55.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.548+0000 7f5847af0640 10 monclient: get_monmap_and_config 2024-09-16T10:31:55.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.548+0000 7f5847af0640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.548+0000 7f5847af0640 10 monclient: monmap: 2024-09-16T10:31:55.550 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.550 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.550 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:55.549592+0000 2024-09-16T10:31:55.550 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:55.549592+0000 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.551 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.549+0000 7f5847af0640 1 Processor -- start 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.549+0000 7f5847af0640 1 -- start start 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.549+0000 7f5847af0640 10 monclient: init 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.549+0000 7f5847af0640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.549+0000 7f5847af0640 10 monclient: _add_conns ranks=[0,6,8,7,1,4,2,3,5] 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f5847af0640 10 monclient(hunting): picked mon.noname-a con 0x7f58400a3730 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f5847af0640 10 monclient(hunting): picked mon.noname-d con 0x7f58400a4220 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f5847af0640 10 monclient(hunting): picked mon.noname-e con 0x7f58400a1780 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f5847af0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5840061ae0 con 0x7f58400a3730 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f5847af0640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58400a0b30 con 0x7f58400a4220 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f5847af0640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58400a1bc0 con 0x7f58400a1780 2024-09-16T10:31:55.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f5847af0640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f5847af0640 10 monclient(hunting): authenticate will time out at 882.890074s 2024-09-16T10:31:55.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f584588c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f58400a3730 0x7f58400a3b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57704/0 (socket says 172.21.15.94:57704) 2024-09-16T10:31:55.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f584588c640 1 -- 172.21.15.94:0/3618113886 learned_addr learned my addr 172.21.15.94:0/3618113886 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.550+0000 7f584608d640 1 --1- 172.21.15.94:0/3618113886 >> v1:172.21.15.94:6793/0 conn(0x7f58400a1780 0x7f584013f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:57964/0 (socket says 172.21.15.94:57964) 2024-09-16T10:31:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584608d640 1 -- 172.21.15.94:0/3618113886 >> v1:172.21.15.94:6793/0 conn(0x7f58400a1780 legacy=0x7f584013f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584588c640 1 -- 172.21.15.94:0/3618113886 >> v1:172.21.15.94:6789/0 conn(0x7f58400a3730 legacy=0x7f58400a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584508b640 1 -- 172.21.15.94:0/3618113886 >> v1:172.21.15.94:6792/0 conn(0x7f58400a4220 legacy=0x7f584013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224808094 0 0) 0x7f5840061ae0 con 0x7f58400a3730 2024-09-16T10:31:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): my global_id is 4617 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f582c003520 con 0x7f58400a3730 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949043429 0 0) 0x7f58400a0b30 con 0x7f58400a4220 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): my global_id is 94236 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5840061ae0 con 0x7f58400a4220 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421742760 0 0) 0x7f58400a1bc0 con 0x7f58400a1780 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient(hunting): my global_id is 4670 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58400a0b30 con 0x7f58400a1780 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896164061 0 0) 0x7f5840061ae0 con 0x7f58400a4220 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 >> v1:172.21.15.94:6793/0 conn(0x7f58400a1780 legacy=0x7f584013f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 >> v1:172.21.15.94:6789/0 conn(0x7f58400a3730 legacy=0x7f58400a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 monclient: found mon.noname-d 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584006ca30 con 0x7f58400a4220 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5830001ab0 con 0x7f58400a4220 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient: dump: 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.564 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f5847af0640 5 monclient: authenticate success, global_id 94236 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f5847af0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5830003610 con 0x7f58400a4220 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f5847af0640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:55.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f5847af0640 4 set_mon_vals no callback set 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.552+0000 7f584488a640 1 -- 172.21.15.94:0/3618113886 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5830003ac0 con 0x7f58400a4220 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f584488a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f584488a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f584488a640 10 monclient: dump: 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f5847af0640 10 monclient: shutdown 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f5847af0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f5847af0640 1 -- 172.21.15.94:0/3618113886 >> v1:172.21.15.94:6792/0 conn(0x7f58400a4220 legacy=0x7f584013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f5847af0640 1 -- 172.21.15.94:0/3618113886 shutdown_connections 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f5847af0640 1 -- 172.21.15.94:0/3618113886 >> 172.21.15.94:0/3618113886 conn(0x7f58400a5830 msgr2=0x7f58400a5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f5847af0640 1 -- 172.21.15.94:0/3618113886 shutdown_connections 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.553+0000 7f5847af0640 1 -- 172.21.15.94:0/3618113886 wait complete. 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient: monmap: 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 1 Processor -- start 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 1 -- start start 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient: init 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient: _add_conns ranks=[8,1,6,0,7,4,2,3,5] 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient(hunting): picked mon.noname-e con 0x7f58400a1780 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient(hunting): picked mon.noname-f con 0x7f58400a3730 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient(hunting): picked mon.noname-d con 0x7f58400a4220 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5840126950 con 0x7f58400a3730 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58401263d0 con 0x7f58400a4220 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584009f040 con 0x7f58400a1780 2024-09-16T10:31:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f5847af0640 10 monclient(hunting): authenticate will time out at 882.894960s 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f584588c640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7f58400a1780 0x7f5840136520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:57970/0 (socket says 172.21.15.94:57970) 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.554+0000 7f584588c640 1 -- 172.21.15.94:0/1030080411 learned_addr learned my addr 172.21.15.94:0/1030080411 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f584588c640 1 -- 172.21.15.94:0/1030080411 >> v1:172.21.15.94:6793/0 conn(0x7f58400a1780 legacy=0x7f5840136520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f584608d640 1 -- 172.21.15.94:0/1030080411 >> v1:172.21.15.94:6792/0 conn(0x7f58400a4220 legacy=0x7f58401d3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183996533 0 0) 0x7f584009f040 con 0x7f58400a1780 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): my global_id is 4679 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5824003520 con 0x7f58400a1780 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942329967 0 0) 0x7f58401263d0 con 0x7f58400a4220 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): my global_id is 94245 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f584009f040 con 0x7f58400a4220 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f584508b640 1 -- 172.21.15.94:0/1030080411 >> v1:172.21.15.152:6789/0 conn(0x7f58400a3730 legacy=0x7f58401cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 221385845 0 0) 0x7f5824003520 con 0x7f58400a1780 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58401263d0 con 0x7f58400a1780 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4104907760 0 0) 0x7f584009f040 con 0x7f58400a4220 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5824003520 con 0x7f58400a4220 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f58380020c0 con 0x7f58400a1780 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): dump: 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.571 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f583c001320 con 0x7f58400a4220 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): dump: 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.573 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496158481 0 0) 0x7f5840126950 con 0x7f58400a3730 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient(hunting): my global_id is 4609 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f584009f040 con 0x7f58400a3730 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 272823987 0 0) 0x7f5824003520 con 0x7f58400a4220 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 >> v1:172.21.15.94:6793/0 conn(0x7f58400a1780 legacy=0x7f5840136520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 >> v1:172.21.15.152:6789/0 conn(0x7f58400a3730 legacy=0x7f58401cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 monclient: found mon.d 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584009ecf0 con 0x7f58400a4220 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f5847af0640 5 monclient: authenticate success, global_id 94245 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f5847af0640 10 monclient: _renew_subs 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f5847af0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5840126c30 con 0x7f58400a4220 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f5847af0640 10 monclient: _renew_subs 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f5847af0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5840136c30 con 0x7f58400a4220 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f583c004a90 con 0x7f58400a4220 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f583c004f40 con 0x7f58400a4220 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f5837fff640 4 set_mon_vals no callback set 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.557+0000 7f58367fc640 10 monclient: dump: 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.558+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f583c0050a0 con 0x7f58400a4220 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.558+0000 7f5847af0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.558+0000 7f5847af0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.558+0000 7f5847af0640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.558+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5828005040 con 0x7f58400a4220 2024-09-16T10:31:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.558+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f583c045ca0 con 0x7f58400a4220 2024-09-16T10:31:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.559+0000 7f584588c640 10 monclient: get_auth_request con 0x7f5824039820 auth_method 0 2024-09-16T10:31:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.563+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f583c0067a0 con 0x7f58400a4220 2024-09-16T10:31:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.563+0000 7f58367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.563+0000 7f58367fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:55.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.724+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5828000b90 con 0x7f5824039820 2024-09-16T10:31:55.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.725+0000 7f58367fc640 1 -- 172.21.15.94:0/1030080411 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (unknown 489605429 0 0) 0x7f5828000b90 con 0x7f5824039820 2024-09-16T10:31:55.727 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T10:31:55.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.727+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 >> v1:172.21.15.152:6800/2686360781 conn(0x7f5824039820 legacy=0x7f582403bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.727+0000 7f5847af0640 10 monclient: shutdown 2024-09-16T10:31:55.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.727+0000 7f5847af0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.727+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 >> v1:172.21.15.94:6792/0 conn(0x7f58400a4220 legacy=0x7f58401d3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.727+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 shutdown_connections 2024-09-16T10:31:55.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.727+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 >> 172.21.15.94:0/1030080411 conn(0x7f58400a5830 msgr2=0x7f58400a6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.727+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 shutdown_connections 2024-09-16T10:31:55.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.727+0000 7f5847af0640 1 -- 172.21.15.94:0/1030080411 wait complete. 2024-09-16T10:31:55.739 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T10:31:55.739 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T10:31:55.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.838+0000 7efeede87640 10 monclient: get_monmap_and_config 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.838+0000 7efeede87640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.838+0000 7efeede87640 10 monclient: monmap: 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:55.838921+0000 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:55.838921+0000 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.840 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.841 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.841 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.841 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.841 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.841 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.841 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.838+0000 7efeede87640 1 Processor -- start 2024-09-16T10:31:55.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 1 -- start start 2024-09-16T10:31:55.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 10 monclient: init 2024-09-16T10:31:55.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 10 monclient: _add_conns ranks=[3,6,7,5,4,1,2,8,0] 2024-09-16T10:31:55.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 10 monclient(hunting): picked mon.noname-g con 0x7efee80a3940 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:55.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 10 monclient(hunting): picked mon.noname-d con 0x7efee806c960 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:55.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 10 monclient(hunting): picked mon.noname-i con 0x7efee80a4430 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:55.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee8061ae0 con 0x7efee80a3940 2024-09-16T10:31:55.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80a6940 con 0x7efee806c960 2024-09-16T10:31:55.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80a06e0 con 0x7efee80a4430 2024-09-16T10:31:55.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efeede87640 10 monclient(hunting): authenticate will time out at 883.179528s 2024-09-16T10:31:55.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efedffff640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7efee806c960 0x7efee813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:33788/0 (socket says 172.21.15.94:33788) 2024-09-16T10:31:55.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.839+0000 7efedffff640 1 -- 172.21.15.94:0/4261213755 learned_addr learned my addr 172.21.15.94:0/4261213755 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.841+0000 7efedffff640 1 -- 172.21.15.94:0/4261213755 >> v1:172.21.15.94:6792/0 conn(0x7efee806c960 legacy=0x7efee813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee7fff640 1 -- 172.21.15.94:0/4261213755 >> v1:172.21.15.152:6792/0 conn(0x7efee80a4430 legacy=0x7efee813f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee77fe640 1 -- 172.21.15.94:0/4261213755 >> v1:172.21.15.152:6790/0 conn(0x7efee80a3940 legacy=0x7efee80a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253757244 0 0) 0x7efee80a6940 con 0x7efee806c960 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): my global_id is 94254 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed4003520 con 0x7efee806c960 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308692097 0 0) 0x7efee8061ae0 con 0x7efee80a3940 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): my global_id is 4890 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee80a6940 con 0x7efee80a3940 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932598584 0 0) 0x7efee80a06e0 con 0x7efee80a4430 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient(hunting): my global_id is 4714 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee8061ae0 con 0x7efee80a4430 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 776856569 0 0) 0x7efed4003520 con 0x7efee806c960 2024-09-16T10:31:55.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 >> v1:172.21.15.152:6792/0 conn(0x7efee80a4430 legacy=0x7efee813f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 >> v1:172.21.15.152:6790/0 conn(0x7efee80a3940 legacy=0x7efee80a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 monclient: found mon.noname-d 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80a09c0 con 0x7efee806c960 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efed0001a60 con 0x7efee806c960 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efeede87640 5 monclient: authenticate success, global_id 94254 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efeede87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: dump: 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.850 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efeede87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efed0001cf0 con 0x7efee806c960 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 1 -- 172.21.15.94:0/4261213755 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efed0004be0 con 0x7efee806c960 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efeede87640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efeede87640 4 set_mon_vals no callback set 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efee6ffd640 10 monclient: dump: 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.851 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.852 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efeede87640 10 monclient: shutdown 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efeede87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.842+0000 7efeede87640 1 -- 172.21.15.94:0/4261213755 >> v1:172.21.15.94:6792/0 conn(0x7efee806c960 legacy=0x7efee813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.843+0000 7efeede87640 1 -- 172.21.15.94:0/4261213755 shutdown_connections 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.843+0000 7efeede87640 1 -- 172.21.15.94:0/4261213755 >> 172.21.15.94:0/4261213755 conn(0x7efee80a5a40 msgr2=0x7efee80a5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.843+0000 7efeede87640 1 -- 172.21.15.94:0/4261213755 shutdown_connections 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.843+0000 7efeede87640 1 -- 172.21.15.94:0/4261213755 wait complete. 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.843+0000 7efeede87640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.843+0000 7efeede87640 10 monclient: monmap: 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.853 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.843+0000 7efeede87640 1 Processor -- start 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.843+0000 7efeede87640 1 -- start start 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.843+0000 7efeede87640 10 monclient: init 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 10 monclient: _add_conns ranks=[5,7,8,2,1,3,4,6,0] 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 10 monclient(hunting): picked mon.noname-h con 0x7efee806c960 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 10 monclient(hunting): picked mon.noname-i con 0x7efee80a3940 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 10 monclient(hunting): picked mon.noname-e con 0x7efee80a4430 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee809ef80 con 0x7efee806c960 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee806ba50 con 0x7efee80a3940 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee8127700 con 0x7efee80a4430 2024-09-16T10:31:55.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efeede87640 10 monclient(hunting): authenticate will time out at 883.184388s 2024-09-16T10:31:55.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efee7fff640 1 --1- >> v1:172.21.15.94:6793/0 conn(0x7efee80a4430 0x7efee81d2df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6793/0 says I am v1:172.21.15.94:57982/0 (socket says 172.21.15.94:57982) 2024-09-16T10:31:55.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.844+0000 7efee7fff640 1 -- 172.21.15.94:0/427343288 learned_addr learned my addr 172.21.15.94:0/427343288 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee7fff640 1 -- 172.21.15.94:0/427343288 >> v1:172.21.15.94:6793/0 conn(0x7efee80a4430 legacy=0x7efee81d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efedffff640 1 -- 172.21.15.94:0/427343288 >> v1:172.21.15.152:6792/0 conn(0x7efee80a3940 legacy=0x7efee8136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee77fe640 1 -- 172.21.15.94:0/427343288 >> v1:172.21.15.152:6791/0 conn(0x7efee806c960 legacy=0x7efee81caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee7fff640 1 --1- 172.21.15.94:0/427343288 >> v1:172.21.15.94:6793/0 conn(0x7efee80a4430 0x7efee81d2df0 :-1 s=OPENED pgs=139 cs=1 l=1).queue_received will delay after 1.72648e+09 on 0x7efee8127700 auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493090155 0 0) 0x7efee809ef80 con 0x7efee806c960 2024-09-16T10:31:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 10 monclient(hunting): my global_id is 4712 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec8003520 con 0x7efee806c960 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079869947 0 0) 0x7efee806ba50 con 0x7efee80a3940 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 10 monclient(hunting): my global_id is 4723 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.846+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee809ef80 con 0x7efee80a3940 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 373558911 0 0) 0x7efec8003520 con 0x7efee806c960 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efee806ba50 con 0x7efee806c960 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efed8001d70 con 0x7efee806c960 2024-09-16T10:31:55.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient(hunting): dump: 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.861 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4198915142 0 0) 0x7efee809ef80 con 0x7efee80a3940 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efec8003520 con 0x7efee80a3940 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efed0004b20 con 0x7efee80a3940 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient(hunting): dump: 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.862 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.863 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.5 v1:172.21.15.152:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 865178496 0 0) 0x7efee806ba50 con 0x7efee806c960 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 >> v1:172.21.15.94:6793/0 conn(0x7efee80a4430 legacy=0x7efee81d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 >> v1:172.21.15.152:6792/0 conn(0x7efee80a3940 legacy=0x7efee8136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 monclient: found mon.h 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.152:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee81258b0 con 0x7efee806c960 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efeede87640 5 monclient: authenticate success, global_id 4712 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004854082 0 0) 0x7efee8127700 con 0x7efee80a4430 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efeede87640 10 monclient: _renew_subs 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efeede87640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.152:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee809f4d0 con 0x7efee806c960 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efeede87640 10 monclient: _renew_subs 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efeede87640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.152:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efee8137420 con 0x7efee806c960 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.5 v1:172.21.15.152:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efed8003b00 con 0x7efee806c960 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.5 v1:172.21.15.152:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7efed8003fb0 con 0x7efee806c960 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee67fc640 4 set_mon_vals no callback set 2024-09-16T10:31:55.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.847+0000 7efee4ff9640 10 monclient: dump: 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.848+0000 7efeede87640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.152:6791/0 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.848+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.152:6791/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efecc000f10 con 0x7efee806c960 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.848+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.5 v1:172.21.15.152:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7efed80041d0 con 0x7efee806c960 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.848+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.5 v1:172.21.15.152:6791/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7efed8045f90 con 0x7efee806c960 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.848+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.94:6805/3102113414 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efec8043cd0 con 0x7efec8040060 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.848+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== mon.5 v1:172.21.15.152:6791/0 9 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (unknown 3265615600 0 0) 0x7efed8046380 con 0x7efee806c960 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.848+0000 7efee4ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.848+0000 7efedffff640 10 monclient: get_auth_request con 0x7efec8039680 auth_method 0 2024-09-16T10:31:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.848+0000 7efee7fff640 10 monclient: get_auth_request con 0x7efec8040060 auth_method 0 2024-09-16T10:31:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.853+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== osd.2 v1:172.21.15.94:6805/3102113414 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7efec8043cd0 con 0x7efec8040060 2024-09-16T10:31:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.865+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 --> v1:172.21.15.94:6805/3102113414 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efecc002b50 con 0x7efec8040060 2024-09-16T10:31:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.865+0000 7efee4ff9640 1 -- 172.21.15.94:0/427343288 <== osd.2 v1:172.21.15.94:6805/3102113414 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7efecc002b50 con 0x7efec8040060 2024-09-16T10:31:55.869 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:55.869 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T10:31:55.869 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 >> v1:172.21.15.94:6805/3102113414 conn(0x7efec8040060 legacy=0x7efec8042480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 >> v1:172.21.15.152:6800/2686360781 conn(0x7efec8039680 legacy=0x7efec803bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efeede87640 10 monclient: shutdown 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efeede87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 >> v1:172.21.15.152:6791/0 conn(0x7efee806c960 legacy=0x7efee81caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efee7fff640 1 -- 172.21.15.94:0/427343288 reap_dead start 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 shutdown_connections 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 >> 172.21.15.94:0/427343288 conn(0x7efee80a5a40 msgr2=0x7efee809ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 shutdown_connections 2024-09-16T10:31:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.866+0000 7efeede87640 1 -- 172.21.15.94:0/427343288 wait complete. 2024-09-16T10:31:55.877 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T10:31:55.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.974+0000 7fe04b69c640 10 monclient: get_monmap_and_config 2024-09-16T10:31:55.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.974+0000 7fe04b69c640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.975+0000 7fe04b69c640 10 monclient: monmap: 2024-09-16T10:31:55.976 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:55.975770+0000 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:55.975770+0000 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.978 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.975+0000 7fe04b69c640 1 Processor -- start 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.975+0000 7fe04b69c640 1 -- start start 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.975+0000 7fe04b69c640 10 monclient: init 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 10 monclient: _add_conns ranks=[3,4,5,0,8,2,1,7,6] 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 10 monclient(hunting): picked mon.noname-g con 0x7fe0440a3860 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 10 monclient(hunting): picked mon.noname-c con 0x7fe04406c1d0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 10 monclient(hunting): picked mon.noname-h con 0x7fe0440a4350 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe044061ae0 con 0x7fe0440a3860 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0440a0600 con 0x7fe04406c1d0 2024-09-16T10:31:55.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0440a0b30 con 0x7fe0440a4350 2024-09-16T10:31:55.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe04b69c640 10 monclient(hunting): authenticate will time out at 883.316315s 2024-09-16T10:31:55.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe048c37640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7fe04406c1d0 0x7fe04413b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:36184/0 (socket says 172.21.15.94:36184) 2024-09-16T10:31:55.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.976+0000 7fe048c37640 1 -- 172.21.15.94:0/4115939122 learned_addr learned my addr 172.21.15.94:0/4115939122 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe048c37640 1 -- 172.21.15.94:0/4115939122 >> v1:172.21.15.94:6791/0 conn(0x7fe04406c1d0 legacy=0x7fe04413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 1 -- 172.21.15.94:0/4115939122 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990317355 0 0) 0x7fe0440a0600 con 0x7fe04406c1d0 2024-09-16T10:31:55.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 10 monclient(hunting): my global_id is 4711 2024-09-16T10:31:55.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 1 -- 172.21.15.94:0/4115939122 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe030003520 con 0x7fe04406c1d0 2024-09-16T10:31:55.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe049c39640 1 -- 172.21.15.94:0/4115939122 >> v1:172.21.15.152:6791/0 conn(0x7fe0440a4350 legacy=0x7fe04413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 1 -- 172.21.15.94:0/4115939122 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 433129494 0 0) 0x7fe030003520 con 0x7fe04406c1d0 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 1 -- 172.21.15.94:0/4115939122 >> v1:172.21.15.152:6791/0 conn(0x7fe0440a4350 legacy=0x7fe04413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe049438640 1 -- 172.21.15.94:0/4115939122 >> v1:172.21.15.152:6790/0 conn(0x7fe0440a3860 legacy=0x7fe0440a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 1 -- 172.21.15.94:0/4115939122 >> v1:172.21.15.152:6790/0 conn(0x7fe0440a3860 legacy=0x7fe0440a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 1 monclient: found mon.noname-c 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 1 -- 172.21.15.94:0/4115939122 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0440a0e10 con 0x7fe04406c1d0 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe04b69c640 5 monclient: authenticate success, global_id 4711 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 1 -- 172.21.15.94:0/4115939122 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe034001b20 con 0x7fe04406c1d0 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe04b69c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.978+0000 7fe03bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe03bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe03bfff640 10 monclient: dump: 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe03bfff640 1 -- 172.21.15.94:0/4115939122 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0340048a0 con 0x7fe04406c1d0 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe03bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe03bfff640 1 -- 172.21.15.94:0/4115939122 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe034004d90 con 0x7fe04406c1d0 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 4 set_mon_vals no callback set 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe03bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe03bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe03bfff640 10 monclient: dump: 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 10 monclient: shutdown 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 1 -- 172.21.15.94:0/4115939122 >> v1:172.21.15.94:6791/0 conn(0x7fe04406c1d0 legacy=0x7fe04413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 1 -- 172.21.15.94:0/4115939122 shutdown_connections 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 1 -- 172.21.15.94:0/4115939122 >> 172.21.15.94:0/4115939122 conn(0x7fe0440a5960 msgr2=0x7fe0440a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 1 -- 172.21.15.94:0/4115939122 shutdown_connections 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.979+0000 7fe04b69c640 1 -- 172.21.15.94:0/4115939122 wait complete. 2024-09-16T10:31:55.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.980+0000 7fe04b69c640 10 monclient: build_initial_monmap 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.980+0000 7fe04b69c640 10 monclient: monmap: 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:55.989 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.980+0000 7fe04b69c640 1 Processor -- start 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.980+0000 7fe04b69c640 1 -- start start 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.980+0000 7fe04b69c640 10 monclient: init 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 10 monclient: _add_conns ranks=[5,0,3,1,2,6,4,7,8] 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 10 monclient(hunting): picked mon.noname-h con 0x7fe04406c1d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 10 monclient(hunting): picked mon.noname-a con 0x7fe0440a3860 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 10 monclient(hunting): picked mon.noname-g con 0x7fe0440a4350 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04409f150 con 0x7fe0440a3860 2024-09-16T10:31:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04406b9c0 con 0x7fe0440a4350 2024-09-16T10:31:55.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0441ac150 con 0x7fe04406c1d0 2024-09-16T10:31:55.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:55.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe04b69c640 10 monclient(hunting): authenticate will time out at 883.321193s 2024-09-16T10:31:55.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe048c37640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe0440a3860 0x7fe0441cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57714/0 (socket says 172.21.15.94:57714) 2024-09-16T10:31:55.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.981+0000 7fe048c37640 1 -- 172.21.15.94:0/3836066418 learned_addr learned my addr 172.21.15.94:0/3836066418 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe048c37640 1 -- 172.21.15.94:0/3836066418 >> v1:172.21.15.94:6789/0 conn(0x7fe0440a3860 legacy=0x7fe0441cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe049c39640 1 -- 172.21.15.94:0/3836066418 >> v1:172.21.15.152:6790/0 conn(0x7fe0440a4350 legacy=0x7fe0441d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444717721 0 0) 0x7fe04409f150 con 0x7fe0440a3860 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): my global_id is 4626 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe049438640 1 -- 172.21.15.94:0/3836066418 >> v1:172.21.15.152:6791/0 conn(0x7fe04406c1d0 legacy=0x7fe0441be790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe028003520 con 0x7fe0440a3860 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007153073 0 0) 0x7fe04406b9c0 con 0x7fe0440a4350 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): my global_id is 4899 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04409f150 con 0x7fe0440a4350 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1492499127 0 0) 0x7fe028003520 con 0x7fe0440a3860 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe04406b9c0 con 0x7fe0440a3860 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe034004d50 con 0x7fe0440a3860 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): dump: 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.995 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287316381 0 0) 0x7fe0441ac150 con 0x7fe04406c1d0 2024-09-16T10:31:55.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _negotiate 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient(hunting): my global_id is 4721 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe028003520 con 0x7fe04406c1d0 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4136711910 0 0) 0x7fe04406b9c0 con 0x7fe0440a3860 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 >> v1:172.21.15.152:6791/0 conn(0x7fe04406c1d0 legacy=0x7fe0441be790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 >> v1:172.21.15.152:6790/0 conn(0x7fe0440a4350 legacy=0x7fe0441d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient: _finish_hunting 0 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 monclient: found mon.a 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04409f6a0 con 0x7fe0440a3860 2024-09-16T10:31:55.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient: _finish_auth 0 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 10 monclient: _check_auth_tickets 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe039ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe04b69c640 5 monclient: authenticate success, global_id 4626 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe04b69c640 10 monclient: _renew_subs 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe04b69c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.983+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0441abe70 con 0x7fe0440a3860 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe034001b00 con 0x7fe0440a3860 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe039ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe0340056b0 con 0x7fe0440a3860 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe039ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe039ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe039ffb640 10 monclient: dump: 2024-09-16T10:31:55.998 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:55.999 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe04b69c640 10 monclient: _renew_subs 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe03b7fe640 4 set_mon_vals no callback set 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe04b69c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0441aa710 con 0x7fe0440a3860 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.984+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fe034010710 con 0x7fe0440a3860 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.985+0000 7fe04b69c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.985+0000 7fe04b69c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.985+0000 7fe04b69c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.985+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7fe0340461f0 con 0x7fe0440a3860 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.985+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0440008d0 con 0x7fe0440a3860 2024-09-16T10:31:56.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.985+0000 7fe049438640 10 monclient: get_auth_request con 0x7fe028039290 auth_method 0 2024-09-16T10:31:56.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.989+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fe034015760 con 0x7fe0440a3860 2024-09-16T10:31:56.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.989+0000 7fe039ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:56.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:55.989+0000 7fe039ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:56.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.156+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0440a4920 con 0x7fe028039290 2024-09-16T10:31:56.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.156+0000 7fe039ffb640 1 -- 172.21.15.94:0/3836066418 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (unknown 3049115190 0 0) 0x7fe040001860 con 0x7fe028039290 2024-09-16T10:31:56.158 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T10:31:56.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.158+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 >> v1:172.21.15.152:6800/2686360781 conn(0x7fe028039290 legacy=0x7fe02803b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.158+0000 7fe04b69c640 10 monclient: shutdown 2024-09-16T10:31:56.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.158+0000 7fe04b69c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:56.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.158+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 >> v1:172.21.15.94:6789/0 conn(0x7fe0440a3860 legacy=0x7fe0441cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.158+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 shutdown_connections 2024-09-16T10:31:56.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.158+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 >> 172.21.15.94:0/3836066418 conn(0x7fe0440a5960 msgr2=0x7fe0440a7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:56.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.158+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 shutdown_connections 2024-09-16T10:31:56.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.158+0000 7fe04b69c640 1 -- 172.21.15.94:0/3836066418 wait complete. 2024-09-16T10:31:56.170 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T10:31:56.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T10:31:56.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.269+0000 7f1510a6b640 10 monclient: get_monmap_and_config 2024-09-16T10:31:56.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.269+0000 7f1510a6b640 10 monclient: build_initial_monmap 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.269+0000 7f1510a6b640 10 monclient: monmap: 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:56.270132+0000 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:56.270132+0000 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:56.271 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:56.272 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:56.272 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:56.272 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:56.272 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.269+0000 7f1510a6b640 1 Processor -- start 2024-09-16T10:31:56.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.269+0000 7f1510a6b640 1 -- start start 2024-09-16T10:31:56.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.269+0000 7f1510a6b640 10 monclient: init 2024-09-16T10:31:56.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:56.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 10 monclient: _add_conns ranks=[3,6,4,5,8,7,1,2,0] 2024-09-16T10:31:56.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 10 monclient(hunting): picked mon.noname-g con 0x7f150c0a38d0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:56.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 10 monclient(hunting): picked mon.noname-d con 0x7f150c0a16a0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:56.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 10 monclient(hunting): picked mon.noname-c con 0x7f150c13c8e0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:56.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c061ae0 con 0x7f150c0a38d0 2024-09-16T10:31:56.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c06c460 con 0x7f150c13c8e0 2024-09-16T10:31:56.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c0a0440 con 0x7f150c0a16a0 2024-09-16T10:31:56.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:56.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1510a6b640 10 monclient(hunting): authenticate will time out at 883.610645s 2024-09-16T10:31:56.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f150ad9d640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f150c13c8e0 0x7f150c13ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:36200/0 (socket says 172.21.15.94:36200) 2024-09-16T10:31:56.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f150ad9d640 1 -- 172.21.15.94:0/3943861988 learned_addr learned my addr 172.21.15.94:0/3943861988 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:56.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.270+0000 7f1509d9b640 1 --1- 172.21.15.94:0/3943861988 >> v1:172.21.15.94:6792/0 conn(0x7f150c0a16a0 0x7f150c0a43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:33802/0 (socket says 172.21.15.94:33802) 2024-09-16T10:31:56.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.272+0000 7f150ad9d640 1 -- 172.21.15.94:0/3943861988 >> v1:172.21.15.94:6791/0 conn(0x7f150c13c8e0 legacy=0x7f150c13ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.272+0000 7f1509d9b640 1 -- 172.21.15.94:0/3943861988 >> v1:172.21.15.94:6792/0 conn(0x7f150c0a16a0 legacy=0x7f150c0a43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330009866 0 0) 0x7f150c0a0440 con 0x7f150c0a16a0 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150a59c640 1 -- 172.21.15.94:0/3943861988 >> v1:172.21.15.152:6790/0 conn(0x7f150c0a38d0 legacy=0x7f150c0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): my global_id is 94263 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f8003520 con 0x7f150c0a16a0 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393325707 0 0) 0x7f150c06c460 con 0x7f150c13c8e0 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): my global_id is 4720 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f150c0a0440 con 0x7f150c13c8e0 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411905998 0 0) 0x7f150c061ae0 con 0x7f150c0a38d0 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient(hunting): my global_id is 4908 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f150c06c460 con 0x7f150c0a38d0 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218478181 0 0) 0x7f150c0a0440 con 0x7f150c13c8e0 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 >> v1:172.21.15.94:6792/0 conn(0x7f150c0a16a0 legacy=0x7f150c0a43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 >> v1:172.21.15.152:6790/0 conn(0x7f150c0a38d0 legacy=0x7f150c0a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: _finish_hunting 0 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 monclient: found mon.noname-c 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c06c8a0 con 0x7f150c13c8e0 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: _finish_auth 0 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: _check_auth_tickets 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f1510a6b640 5 monclient: authenticate success, global_id 4720 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1504001d60 con 0x7f150c13c8e0 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f1510a6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: dump: 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.285 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f1510a6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1504002070 con 0x7f150c13c8e0 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 1 -- 172.21.15.94:0/3943861988 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1504004f60 con 0x7f150c13c8e0 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f1510a6b640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f1510a6b640 4 set_mon_vals no callback set 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.273+0000 7f150959a640 10 monclient: dump: 2024-09-16T10:31:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.287 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.274+0000 7f1510a6b640 10 monclient: shutdown 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.274+0000 7f1510a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.274+0000 7f1510a6b640 1 -- 172.21.15.94:0/3943861988 >> v1:172.21.15.94:6791/0 conn(0x7f150c13c8e0 legacy=0x7f150c13ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.274+0000 7f1510a6b640 1 -- 172.21.15.94:0/3943861988 shutdown_connections 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.274+0000 7f1510a6b640 1 -- 172.21.15.94:0/3943861988 >> 172.21.15.94:0/3943861988 conn(0x7f150c0a9500 msgr2=0x7f150c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.274+0000 7f1510a6b640 1 -- 172.21.15.94:0/3943861988 shutdown_connections 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.274+0000 7f1510a6b640 1 -- 172.21.15.94:0/3943861988 wait complete. 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.274+0000 7f1510a6b640 10 monclient: build_initial_monmap 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.274+0000 7f1510a6b640 10 monclient: monmap: 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 1 Processor -- start 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 1 -- start start 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 10 monclient: init 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 10 monclient: _add_conns ranks=[1,8,6,5,7,3,2,4,0] 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 10 monclient(hunting): picked mon.noname-f con 0x7f150c0a16a0 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 10 monclient(hunting): picked mon.noname-e con 0x7f150c0a38d0 addr v1:172.21.15.94:6793/0 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 10 monclient(hunting): picked mon.noname-d con 0x7f150c13c8e0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c13fd80 con 0x7f150c0a16a0 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c0a6f40 con 0x7f150c13c8e0 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c0a5c40 con 0x7f150c0a38d0 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f1510a6b640 10 monclient(hunting): authenticate will time out at 883.615806s 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f150ad9d640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f150c13c8e0 0x7f150c1d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:33804/0 (socket says 172.21.15.94:33804) 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.275+0000 7f150ad9d640 1 -- 172.21.15.94:0/3916635923 learned_addr learned my addr 172.21.15.94:0/3916635923 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:56.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.277+0000 7f150ad9d640 1 -- 172.21.15.94:0/3916635923 >> v1:172.21.15.94:6792/0 conn(0x7f150c13c8e0 legacy=0x7f150c1d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.277+0000 7f1509d9b640 1 -- 172.21.15.94:0/3916635923 >> v1:172.21.15.94:6793/0 conn(0x7f150c0a38d0 legacy=0x7f150c1cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f150a59c640 1 -- 172.21.15.94:0/3916635923 >> v1:172.21.15.152:6789/0 conn(0x7f150c0a16a0 legacy=0x7f150c13bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561476341 0 0) 0x7f150c0a6f40 con 0x7f150c13c8e0 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): my global_id is 94272 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14ec003520 con 0x7f150c13c8e0 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663495584 0 0) 0x7f150c0a5c40 con 0x7f150c0a38d0 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): my global_id is 4697 2024-09-16T10:31:56.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f150c0a6f40 con 0x7f150c0a38d0 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088446708 0 0) 0x7f150c13fd80 con 0x7f150c0a16a0 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): my global_id is 4618 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f150c0a5c40 con 0x7f150c0a16a0 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 190788346 0 0) 0x7f14ec003520 con 0x7f150c13c8e0 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f150c13fd80 con 0x7f150c13c8e0 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1504004ea0 con 0x7f150c13c8e0 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): dump: 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.292 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.293 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1198870752 0 0) 0x7f150c0a6f40 con 0x7f150c0a38d0 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14ec003520 con 0x7f150c0a38d0 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f14f4001960 con 0x7f150c0a38d0 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient(hunting): dump: 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.294 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4156978822 0 0) 0x7f150c13fd80 con 0x7f150c13c8e0 2024-09-16T10:31:56.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 >> v1:172.21.15.94:6793/0 conn(0x7f150c0a38d0 legacy=0x7f150c1cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 >> v1:172.21.15.152:6789/0 conn(0x7f150c0a16a0 legacy=0x7f150c13bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 monclient: found mon.d 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c123800 con 0x7f150c13c8e0 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient: _finish_auth 0 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1510a6b640 5 monclient: authenticate success, global_id 94272 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1510a6b640 10 monclient: _renew_subs 2024-09-16T10:31:56.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3648250811 0 0) 0x7f150c0a5c40 con 0x7f150c0a16a0 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1510a6b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c13c5b0 con 0x7f150c13c8e0 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1510a6b640 10 monclient: _renew_subs 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1510a6b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f150c122290 con 0x7f150c13c8e0 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15040052b0 con 0x7f150c13c8e0 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f1504005760 con 0x7f150c13c8e0 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1508d99640 4 set_mon_vals no callback set 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.278+0000 7f1502ffd640 10 monclient: dump: 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.298 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.279+0000 7f1510a6b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.279+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.94:6792/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14f0000f10 con 0x7f150c13c8e0 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.279+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f15040107c0 con 0x7f150c13c8e0 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.279+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f1504046200 con 0x7f150c13c8e0 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.279+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.152:6805/823649059 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14ec0440d0 con 0x7f14ec040460 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.279+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (unknown 3265615600 0 0) 0x7f15040465f0 con 0x7f150c13c8e0 2024-09-16T10:31:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.279+0000 7f1502ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2024-09-16T10:31:56.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.280+0000 7f150a59c640 10 monclient: get_auth_request con 0x7f14ec039870 auth_method 0 2024-09-16T10:31:56.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.280+0000 7f1509d9b640 10 monclient: get_auth_request con 0x7f14ec040460 auth_method 0 2024-09-16T10:31:56.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.285+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== osd.3 v1:172.21.15.152:6805/823649059 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f14ec0440d0 con 0x7f14ec040460 2024-09-16T10:31:56.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.297+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 --> v1:172.21.15.152:6805/823649059 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f14f0002b50 con 0x7f14ec040460 2024-09-16T10:31:56.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1502ffd640 1 -- 172.21.15.94:0/3916635923 <== osd.3 v1:172.21.15.152:6805/823649059 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f14f0002b50 con 0x7f14ec040460 2024-09-16T10:31:56.301 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T10:31:56.301 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:56.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 >> v1:172.21.15.152:6805/823649059 conn(0x7f14ec040460 legacy=0x7f14ec042880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 >> v1:172.21.15.152:6800/2686360781 conn(0x7f14ec039870 legacy=0x7f14ec03bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1510a6b640 10 monclient: shutdown 2024-09-16T10:31:56.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1510a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:56.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 >> v1:172.21.15.94:6792/0 conn(0x7f150c13c8e0 legacy=0x7f150c1d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f150ad9d640 1 -- 172.21.15.94:0/3916635923 reap_dead start 2024-09-16T10:31:56.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 shutdown_connections 2024-09-16T10:31:56.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 >> 172.21.15.94:0/3916635923 conn(0x7f150c0a9500 msgr2=0x7f150c0a1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:56.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 shutdown_connections 2024-09-16T10:31:56.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.298+0000 7f1510a6b640 1 -- 172.21.15.94:0/3916635923 wait complete. 2024-09-16T10:31:56.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T10:31:56.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.405+0000 7f675be90640 10 monclient: get_monmap_and_config 2024-09-16T10:31:56.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.405+0000 7f675be90640 10 monclient: build_initial_monmap 2024-09-16T10:31:56.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.405+0000 7f675be90640 10 monclient: monmap: 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:56.405928+0000 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:56.405928+0000 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:56.407 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:56.408 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:56.408 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:56.408 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:56.408 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.405+0000 7f675be90640 1 Processor -- start 2024-09-16T10:31:56.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.405+0000 7f675be90640 1 -- start start 2024-09-16T10:31:56.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.405+0000 7f675be90640 10 monclient: init 2024-09-16T10:31:56.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:56.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 10 monclient: _add_conns ranks=[5,4,7,2,1,6,3,8,0] 2024-09-16T10:31:56.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 10 monclient(hunting): picked mon.noname-h con 0x7f67540a38d0 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:56.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 10 monclient(hunting): picked mon.noname-c con 0x7f675406c670 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:56.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 10 monclient(hunting): picked mon.noname-i con 0x7f67540a43c0 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:56.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6754061ae0 con 0x7f675406c670 2024-09-16T10:31:56.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67540a04d0 con 0x7f67540a38d0 2024-09-16T10:31:56.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67540a3250 con 0x7f67540a43c0 2024-09-16T10:31:56.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:56.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675be90640 10 monclient(hunting): authenticate will time out at 883.746504s 2024-09-16T10:31:56.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675942b640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f675406c670 0x7f675413bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:36212/0 (socket says 172.21.15.94:36212) 2024-09-16T10:31:56.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.406+0000 7f675942b640 1 -- 172.21.15.94:0/1223623015 learned_addr learned my addr 172.21.15.94:0/1223623015 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.408+0000 7f675942b640 1 -- 172.21.15.94:0/1223623015 >> v1:172.21.15.94:6791/0 conn(0x7f675406c670 legacy=0x7f675413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6759c2c640 1 -- 172.21.15.94:0/1223623015 >> v1:172.21.15.152:6791/0 conn(0x7f67540a38d0 legacy=0x7f67540a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f675a42d640 1 -- 172.21.15.94:0/1223623015 >> v1:172.21.15.152:6792/0 conn(0x7f67540a43c0 legacy=0x7f675413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177380722 0 0) 0x7f6754061ae0 con 0x7f675406c670 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): my global_id is 4729 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6740003520 con 0x7f675406c670 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087081201 0 0) 0x7f67540a04d0 con 0x7f67540a38d0 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): my global_id is 4730 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6754061ae0 con 0x7f67540a38d0 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396512195 0 0) 0x7f67540a3250 con 0x7f67540a43c0 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient(hunting): my global_id is 4732 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67540a04d0 con 0x7f67540a43c0 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2056557103 0 0) 0x7f6740003520 con 0x7f675406c670 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 >> v1:172.21.15.152:6792/0 conn(0x7f67540a43c0 legacy=0x7f675413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 >> v1:172.21.15.152:6791/0 conn(0x7f67540a38d0 legacy=0x7f67540a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: _finish_hunting 0 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 monclient: found mon.noname-c 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.94:6791/0 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67540a0a00 con 0x7f675406c670 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: _finish_auth 0 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: _check_auth_tickets 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f675be90640 5 monclient: authenticate success, global_id 4729 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f675be90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6744001a60 con 0x7f675406c670 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:56.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: dump: 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f675be90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 <== mon.4 v1:172.21.15.94:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6744001d70 con 0x7f675406c670 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f675be90640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 1 -- 172.21.15.94:0/1223623015 <== mon.4 v1:172.21.15.94:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6744004be0 con 0x7f675406c670 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f675be90640 4 set_mon_vals no callback set 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.409+0000 7f6758c2a640 10 monclient: dump: 2024-09-16T10:31:56.418 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.410+0000 7f675be90640 10 monclient: shutdown 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.410+0000 7f675be90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.410+0000 7f675be90640 1 -- 172.21.15.94:0/1223623015 >> v1:172.21.15.94:6791/0 conn(0x7f675406c670 legacy=0x7f675413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.410+0000 7f675be90640 1 -- 172.21.15.94:0/1223623015 shutdown_connections 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.410+0000 7f675be90640 1 -- 172.21.15.94:0/1223623015 >> 172.21.15.94:0/1223623015 conn(0x7f67540a59d0 msgr2=0x7f67540a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.410+0000 7f675be90640 1 -- 172.21.15.94:0/1223623015 shutdown_connections 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.410+0000 7f675be90640 1 -- 172.21.15.94:0/1223623015 wait complete. 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.410+0000 7f675be90640 10 monclient: build_initial_monmap 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.410+0000 7f675be90640 10 monclient: monmap: 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:56.420 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 1 Processor -- start 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 1 -- start start 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 10 monclient: init 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 10 monclient: _add_conns ranks=[5,1,6,4,7,2,3,8,0] 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 10 monclient(hunting): picked mon.noname-h con 0x7f675406c670 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 10 monclient(hunting): picked mon.noname-f con 0x7f67540a38d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 10 monclient(hunting): picked mon.noname-d con 0x7f67540a43c0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675406ba70 con 0x7f67540a38d0 2024-09-16T10:31:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6754125550 con 0x7f675406c670 2024-09-16T10:31:56.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6754126e80 con 0x7f67540a43c0 2024-09-16T10:31:56.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:56.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675be90640 10 monclient(hunting): authenticate will time out at 883.751509s 2024-09-16T10:31:56.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675a42d640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f67540a43c0 0x7f67541d3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:33818/0 (socket says 172.21.15.94:33818) 2024-09-16T10:31:56.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.411+0000 7f675a42d640 1 -- 172.21.15.94:0/2111239135 learned_addr learned my addr 172.21.15.94:0/2111239135 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:56.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f675a42d640 1 -- 172.21.15.94:0/2111239135 >> v1:172.21.15.94:6792/0 conn(0x7f67540a43c0 legacy=0x7f67541d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f6759c2c640 1 -- 172.21.15.94:0/2111239135 >> v1:172.21.15.152:6791/0 conn(0x7f675406c670 legacy=0x7f6754136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f675942b640 1 -- 172.21.15.94:0/2111239135 >> v1:172.21.15.152:6789/0 conn(0x7f67540a38d0 legacy=0x7f67541cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168560884 0 0) 0x7f6754126e80 con 0x7f67540a43c0 2024-09-16T10:31:56.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f674a7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f674a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f674a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f674a7fc640 10 monclient(hunting): my global_id is 94281 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6738003520 con 0x7f67540a43c0 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.413+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545308160 0 0) 0x7f6754125550 con 0x7f675406c670 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): my global_id is 4739 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6754126e80 con 0x7f675406c670 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181337342 0 0) 0x7f675406ba70 con 0x7f67540a38d0 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): my global_id is 4627 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6754125550 con 0x7f67540a38d0 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2245643570 0 0) 0x7f6738003520 con 0x7f67540a43c0 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f675406ba70 con 0x7f67540a43c0 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f674c0020c0 con 0x7f67540a43c0 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): dump: 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3302131986 0 0) 0x7f6754126e80 con 0x7f675406c670 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6738003520 con 0x7f675406c670 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f6750001d70 con 0x7f675406c670 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient(hunting): dump: 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.430 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1585671392 0 0) 0x7f675406ba70 con 0x7f67540a43c0 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 >> v1:172.21.15.152:6791/0 conn(0x7f675406c670 legacy=0x7f6754136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 >> v1:172.21.15.152:6789/0 conn(0x7f67540a38d0 legacy=0x7f67541cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:56.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 monclient: found mon.d 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6754124d90 con 0x7f67540a43c0 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f675be90640 5 monclient: authenticate success, global_id 94281 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f675be90640 10 monclient: _renew_subs 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f675be90640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675409f290 con 0x7f67540a43c0 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f675be90640 10 monclient: _renew_subs 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f675be90640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6754126ba0 con 0x7f67540a43c0 2024-09-16T10:31:56.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f674c003e30 con 0x7f67540a43c0 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f674c0042e0 con 0x7f67540a43c0 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674a7fc640 10 monclient: dump: 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.414+0000 7f674bfff640 4 set_mon_vals no callback set 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.415+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f674c004440 con 0x7f67540a43c0 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.415+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f674c046200 con 0x7f67540a43c0 2024-09-16T10:31:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.415+0000 7f675be90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.415+0000 7f675be90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.415+0000 7f675be90640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.415+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67541269a0 con 0x7f67540a43c0 2024-09-16T10:31:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.415+0000 7f6759c2c640 10 monclient: get_auth_request con 0x7f6738039980 auth_method 0 2024-09-16T10:31:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.420+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f674c015500 con 0x7f67540a43c0 2024-09-16T10:31:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.420+0000 7f674a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.420+0000 7f674a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:56.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.579+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f675404e730 con 0x7f6738039980 2024-09-16T10:31:56.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.580+0000 7f674a7fc640 1 -- 172.21.15.94:0/2111239135 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (unknown 3526570807 0 0) 0x7f6750003aa0 con 0x7f6738039980 2024-09-16T10:31:56.581 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T10:31:56.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.582+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 >> v1:172.21.15.152:6800/2686360781 conn(0x7f6738039980 legacy=0x7f673803be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.582+0000 7f675be90640 10 monclient: shutdown 2024-09-16T10:31:56.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.582+0000 7f675be90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:56.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.582+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 >> v1:172.21.15.94:6792/0 conn(0x7f67540a43c0 legacy=0x7f67541d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.582+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 shutdown_connections 2024-09-16T10:31:56.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.582+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 >> 172.21.15.94:0/2111239135 conn(0x7f67540a59d0 msgr2=0x7f675409ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:56.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.582+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 shutdown_connections 2024-09-16T10:31:56.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.582+0000 7f675be90640 1 -- 172.21.15.94:0/2111239135 wait complete. 2024-09-16T10:31:56.594 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T10:31:56.594 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T10:31:56.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.692+0000 7f98942e9640 10 monclient: get_monmap_and_config 2024-09-16T10:31:56.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.692+0000 7f98942e9640 10 monclient: build_initial_monmap 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.692+0000 7f98942e9640 10 monclient: monmap: 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:56.692956+0000 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:56.692956+0000 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:56.694 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:56.695 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:56.695 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:56.695 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:56.695 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:56.695 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:56.695 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.692+0000 7f98942e9640 1 Processor -- start 2024-09-16T10:31:56.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.692+0000 7f98942e9640 1 -- start start 2024-09-16T10:31:56.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.692+0000 7f98942e9640 10 monclient: init 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 10 monclient: _add_conns ranks=[3,1,6,2,5,0,7,4,8] 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 10 monclient(hunting): picked mon.noname-g con 0x7f988c0a3940 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 10 monclient(hunting): picked mon.noname-f con 0x7f988c06c960 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 10 monclient(hunting): picked mon.noname-d con 0x7f988c0a4430 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c061ae0 con 0x7f988c06c960 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c0a6940 con 0x7f988c0a3940 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c0a06e0 con 0x7f988c0a4430 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f98942e9640 10 monclient(hunting): authenticate will time out at 884.033530s 2024-09-16T10:31:56.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f9892886640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f988c0a4430 0x7f988c13f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:33830/0 (socket says 172.21.15.94:33830) 2024-09-16T10:31:56.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.693+0000 7f9892886640 1 -- 172.21.15.94:0/759241490 learned_addr learned my addr 172.21.15.94:0/759241490 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:56.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.695+0000 7f9892886640 1 -- 172.21.15.94:0/759241490 >> v1:172.21.15.94:6792/0 conn(0x7f988c0a4430 legacy=0x7f988c13f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 -- 172.21.15.94:0/759241490 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224373695 0 0) 0x7f988c0a06e0 con 0x7f988c0a4430 2024-09-16T10:31:56.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient(hunting): my global_id is 94290 2024-09-16T10:31:56.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 -- 172.21.15.94:0/759241490 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9878003520 con 0x7f988c0a4430 2024-09-16T10:31:56.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891884640 1 -- 172.21.15.94:0/759241490 >> v1:172.21.15.152:6789/0 conn(0x7f988c06c960 legacy=0x7f988c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9892085640 1 -- 172.21.15.94:0/759241490 >> v1:172.21.15.152:6790/0 conn(0x7f988c0a3940 legacy=0x7f988c0a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 -- 172.21.15.94:0/759241490 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2889945494 0 0) 0x7f9878003520 con 0x7f988c0a4430 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 -- 172.21.15.94:0/759241490 >> v1:172.21.15.152:6790/0 conn(0x7f988c0a3940 legacy=0x7f988c0a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 -- 172.21.15.94:0/759241490 >> v1:172.21.15.152:6789/0 conn(0x7f988c06c960 legacy=0x7f988c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: _finish_hunting 0 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 monclient: found mon.noname-d 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 -- 172.21.15.94:0/759241490 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0a09c0 con 0x7f988c0a4430 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: _finish_auth 0 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: _check_auth_tickets 2024-09-16T10:31:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 5 monclient: authenticate success, global_id 94290 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 -- 172.21.15.94:0/759241490 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9884001de0 con 0x7f988c0a4430 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: dump: 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.708 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.709 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 -- 172.21.15.94:0/759241490 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98840020f0 con 0x7f988c0a4430 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 1 -- 172.21.15.94:0/759241490 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9884004fa0 con 0x7f988c0a4430 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 4 set_mon_vals no callback set 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:56.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f9891083640 10 monclient: dump: 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.711 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 10 monclient: shutdown 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 1 -- 172.21.15.94:0/759241490 >> v1:172.21.15.94:6792/0 conn(0x7f988c0a4430 legacy=0x7f988c13f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 1 -- 172.21.15.94:0/759241490 shutdown_connections 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.696+0000 7f98942e9640 1 -- 172.21.15.94:0/759241490 >> 172.21.15.94:0/759241490 conn(0x7f988c0a5a40 msgr2=0x7f988c0a5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:56.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.697+0000 7f98942e9640 1 -- 172.21.15.94:0/759241490 shutdown_connections 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.697+0000 7f98942e9640 1 -- 172.21.15.94:0/759241490 wait complete. 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.697+0000 7f98942e9640 10 monclient: build_initial_monmap 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.697+0000 7f98942e9640 10 monclient: monmap: 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.713 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.697+0000 7f98942e9640 1 Processor -- start 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.697+0000 7f98942e9640 1 -- start start 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.697+0000 7f98942e9640 10 monclient: init 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 10 monclient: _add_conns ranks=[3,1,7,4,5,0,2,8,6] 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 10 monclient(hunting): picked mon.noname-g con 0x7f988c06c960 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 10 monclient(hunting): picked mon.noname-f con 0x7f988c0a3940 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 10 monclient(hunting): picked mon.noname-i con 0x7f988c0a4430 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c09ecc0 con 0x7f988c0a3940 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c06bae0 con 0x7f988c06c960 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c127660 con 0x7f988c0a4430 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f98942e9640 10 monclient(hunting): authenticate will time out at 884.038353s 2024-09-16T10:31:56.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f9891884640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7f988c0a3940 0x7f988c136c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.94:49168/0 (socket says 172.21.15.94:49168) 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f9892886640 1 --1- >> v1:172.21.15.152:6792/0 conn(0x7f988c0a4430 0x7f988c1d2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6792/0 says I am v1:172.21.15.94:37524/0 (socket says 172.21.15.94:37524) 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f9892085640 1 --1- >> v1:172.21.15.152:6790/0 conn(0x7f988c06c960 0x7f988c1caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6790/0 says I am v1:172.21.15.94:38620/0 (socket says 172.21.15.94:38620) 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.698+0000 7f9891884640 1 -- 172.21.15.94:0/1878702887 learned_addr learned my addr 172.21.15.94:0/1878702887 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9892085640 1 -- 172.21.15.94:0/1878702887 >> v1:172.21.15.152:6790/0 conn(0x7f988c06c960 legacy=0x7f988c1caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9892886640 1 -- 172.21.15.94:0/1878702887 >> v1:172.21.15.152:6792/0 conn(0x7f988c0a4430 legacy=0x7f988c1d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9891884640 1 -- 172.21.15.94:0/1878702887 >> v1:172.21.15.152:6789/0 conn(0x7f988c0a3940 legacy=0x7f988c136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261270079 0 0) 0x7f988c06bae0 con 0x7f988c06c960 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): my global_id is 4926 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9870003520 con 0x7f988c06c960 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248351649 0 0) 0x7f988c09ecc0 con 0x7f988c0a3940 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): my global_id is 4645 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c06bae0 con 0x7f988c0a3940 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781228180 0 0) 0x7f988c127660 con 0x7f988c0a4430 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 10 monclient(hunting): my global_id is 4741 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.700+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c09ecc0 con 0x7f988c0a4430 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1294665672 0 0) 0x7f9870003520 con 0x7f988c06c960 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f988c127660 con 0x7f988c06c960 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9888001d50 con 0x7f988c06c960 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): dump: 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.718 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3565633244 0 0) 0x7f988c06bae0 con 0x7f988c0a3940 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9870003520 con 0x7f988c0a3940 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f987c003610 con 0x7f988c0a3940 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): dump: 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.721 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.722 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.722 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.722 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.7 v1:172.21.15.152:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2887784851 0 0) 0x7f988c09ecc0 con 0x7f988c0a4430 2024-09-16T10:31:56.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f988c06bae0 con 0x7f988c0a4430 2024-09-16T10:31:56.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.7 v1:172.21.15.152:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f98840016a0 con 0x7f988c0a4430 2024-09-16T10:31:56.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient(hunting): dump: 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.723 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.3 v1:172.21.15.152:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 124501059 0 0) 0x7f988c127660 con 0x7f988c06c960 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 >> v1:172.21.15.152:6792/0 conn(0x7f988c0a4430 legacy=0x7f988c1d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 >> v1:172.21.15.152:6789/0 conn(0x7f988c0a3940 legacy=0x7f988c136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient: _finish_hunting 0 2024-09-16T10:31:56.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 monclient: found mon.g 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c125810 con 0x7f988c06c960 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient: _finish_auth 0 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient: _check_auth_tickets 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f98942e9640 5 monclient: authenticate success, global_id 4926 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f98942e9640 10 monclient: _renew_subs 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f98942e9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c09f210 con 0x7f988c06c960 2024-09-16T10:31:56.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f98942e9640 10 monclient: _renew_subs 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f98942e9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c125300 con 0x7f988c06c960 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.3 v1:172.21.15.152:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9888003b10 con 0x7f988c06c960 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.3 v1:172.21.15.152:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9888003fc0 con 0x7f988c06c960 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9890882640 4 set_mon_vals no callback set 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.701+0000 7f9882ffd640 10 monclient: dump: 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.727 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.702+0000 7f98942e9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.152:6790/0 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.702+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9874000f10 con 0x7f988c06c960 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.702+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.3 v1:172.21.15.152:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f98880041a0 con 0x7f988c06c960 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.702+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.3 v1:172.21.15.152:6790/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f9888045fa0 con 0x7f988c06c960 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.702+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6801/1475699231 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f98700443c0 con 0x7f9870040750 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.702+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== mon.3 v1:172.21.15.152:6790/0 9 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (unknown 3265615600 0 0) 0x7f98880462a0 con 0x7f988c06c960 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.702+0000 7f9882ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2024-09-16T10:31:56.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.702+0000 7f9891884640 10 monclient: get_auth_request con 0x7f9870039cd0 auth_method 0 2024-09-16T10:31:56.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.703+0000 7f9892886640 10 monclient: get_auth_request con 0x7f9870040750 auth_method 0 2024-09-16T10:31:56.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.708+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== osd.4 v1:172.21.15.152:6801/1475699231 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f98700443c0 con 0x7f9870040750 2024-09-16T10:31:56.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.719+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 --> v1:172.21.15.152:6801/1475699231 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9874002b20 con 0x7f9870040750 2024-09-16T10:31:56.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f9882ffd640 1 -- 172.21.15.94:0/1878702887 <== osd.4 v1:172.21.15.152:6801/1475699231 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f9874002b20 con 0x7f9870040750 2024-09-16T10:31:56.730 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T10:31:56.730 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:56.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 >> v1:172.21.15.152:6801/1475699231 conn(0x7f9870040750 legacy=0x7f9870042b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 >> v1:172.21.15.152:6800/2686360781 conn(0x7f9870039cd0 legacy=0x7f987003c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f98942e9640 10 monclient: shutdown 2024-09-16T10:31:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f98942e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 >> v1:172.21.15.152:6790/0 conn(0x7f988c06c960 legacy=0x7f988c1caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f9892886640 1 -- 172.21.15.94:0/1878702887 reap_dead start 2024-09-16T10:31:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 shutdown_connections 2024-09-16T10:31:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 >> 172.21.15.94:0/1878702887 conn(0x7f988c0a5a40 msgr2=0x7f988c0a8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 shutdown_connections 2024-09-16T10:31:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.720+0000 7f98942e9640 1 -- 172.21.15.94:0/1878702887 wait complete. 2024-09-16T10:31:56.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T10:31:56.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.830+0000 7fe4152c9640 10 monclient: get_monmap_and_config 2024-09-16T10:31:56.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.830+0000 7fe4152c9640 10 monclient: build_initial_monmap 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.830+0000 7fe4152c9640 10 monclient: monmap: 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:56.830900+0000 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:56.830900+0000 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:56.833 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.830+0000 7fe4152c9640 1 Processor -- start 2024-09-16T10:31:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.830+0000 7fe4152c9640 1 -- start start 2024-09-16T10:31:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.830+0000 7fe4152c9640 10 monclient: init 2024-09-16T10:31:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 10 monclient: _add_conns ranks=[1,0,2,3,6,5,8,4,7] 2024-09-16T10:31:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 10 monclient(hunting): picked mon.noname-f con 0x7fe4100a38d0 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe41006c670 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe4100a43c0 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe410061ae0 con 0x7fe41006c670 2024-09-16T10:31:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4100a04d0 con 0x7fe4100a38d0 2024-09-16T10:31:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4100a3250 con 0x7fe4100a43c0 2024-09-16T10:31:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe4152c9640 10 monclient(hunting): authenticate will time out at 884.171490s 2024-09-16T10:31:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe407fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe41006c670 0x7fe41013bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57726/0 (socket says 172.21.15.94:57726) 2024-09-16T10:31:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe407fff640 1 -- 172.21.15.94:0/981127860 learned_addr learned my addr 172.21.15.94:0/981127860 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.831+0000 7fe40f7fe640 1 --1- 172.21.15.94:0/981127860 >> v1:172.21.15.94:6790/0 conn(0x7fe4100a43c0 0x7fe41013f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:46154/0 (socket says 172.21.15.94:46154) 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40f7fe640 1 -- 172.21.15.94:0/981127860 >> v1:172.21.15.94:6790/0 conn(0x7fe4100a43c0 legacy=0x7fe41013f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe407fff640 1 -- 172.21.15.94:0/981127860 >> v1:172.21.15.94:6789/0 conn(0x7fe41006c670 legacy=0x7fe41013bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241026479 0 0) 0x7fe410061ae0 con 0x7fe41006c670 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 10 monclient(hunting): my global_id is 4635 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3fc003520 con 0x7fe41006c670 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9636560 0 0) 0x7fe4100a3250 con 0x7fe4100a43c0 2024-09-16T10:31:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 10 monclient(hunting): my global_id is 4646 2024-09-16T10:31:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.833+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe410061ae0 con 0x7fe4100a43c0 2024-09-16T10:31:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40effd640 1 -- 172.21.15.94:0/981127860 >> v1:172.21.15.152:6789/0 conn(0x7fe4100a38d0 legacy=0x7fe4100a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 135762340 0 0) 0x7fe3fc003520 con 0x7fe41006c670 2024-09-16T10:31:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 >> v1:172.21.15.94:6790/0 conn(0x7fe4100a43c0 legacy=0x7fe41013f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 >> v1:172.21.15.152:6789/0 conn(0x7fe4100a38d0 legacy=0x7fe4100a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 1 monclient: found mon.noname-a 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4100a0a00 con 0x7fe41006c670 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe3f8001a60 con 0x7fe41006c670 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 5 monclient: authenticate success, global_id 4635 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: dump: 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3f8001d70 con 0x7fe41006c670 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 1 -- 172.21.15.94:0/981127860 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe3f8004be0 con 0x7fe41006c670 2024-09-16T10:31:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 4 set_mon_vals no callback set 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe40e7fc640 10 monclient: dump: 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 10 monclient: shutdown 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:56.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 1 -- 172.21.15.94:0/981127860 >> v1:172.21.15.94:6789/0 conn(0x7fe41006c670 legacy=0x7fe41013bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 1 -- 172.21.15.94:0/981127860 shutdown_connections 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 1 -- 172.21.15.94:0/981127860 >> 172.21.15.94:0/981127860 conn(0x7fe4100a59d0 msgr2=0x7fe4100a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 1 -- 172.21.15.94:0/981127860 shutdown_connections 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.834+0000 7fe4152c9640 1 -- 172.21.15.94:0/981127860 wait complete. 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.835+0000 7fe4152c9640 10 monclient: build_initial_monmap 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.835+0000 7fe4152c9640 10 monclient: monmap: 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:56.849 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.835+0000 7fe4152c9640 1 Processor -- start 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.835+0000 7fe4152c9640 1 -- start start 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.835+0000 7fe4152c9640 10 monclient: init 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 10 monclient: _add_conns ranks=[5,2,3,8,4,0,7,6,1] 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 10 monclient(hunting): picked mon.noname-h con 0x7fe41006c670 addr v1:172.21.15.152:6791/0 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe4100a38d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 10 monclient(hunting): picked mon.noname-g con 0x7fe4100a43c0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe410122a50 con 0x7fe4100a38d0 2024-09-16T10:31:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe410132fa0 con 0x7fe4100a43c0 2024-09-16T10:31:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4101224d0 con 0x7fe41006c670 2024-09-16T10:31:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe4152c9640 10 monclient(hunting): authenticate will time out at 884.176256s 2024-09-16T10:31:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe407fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fe4100a38d0 0x7fe4101cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:46156/0 (socket says 172.21.15.94:46156) 2024-09-16T10:31:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.836+0000 7fe407fff640 1 -- 172.21.15.94:0/695162154 learned_addr learned my addr 172.21.15.94:0/695162154 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:56.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe407fff640 1 -- 172.21.15.94:0/695162154 >> v1:172.21.15.94:6790/0 conn(0x7fe4100a38d0 legacy=0x7fe4101cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867123186 0 0) 0x7fe410122a50 con 0x7fe4100a38d0 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): my global_id is 4655 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f0003520 con 0x7fe4100a38d0 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe40effd640 1 -- 172.21.15.94:0/695162154 >> v1:172.21.15.152:6791/0 conn(0x7fe41006c670 legacy=0x7fe410132890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe40f7fe640 1 -- 172.21.15.94:0/695162154 >> v1:172.21.15.152:6790/0 conn(0x7fe4100a43c0 legacy=0x7fe4101d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2422555270 0 0) 0x7fe3f0003520 con 0x7fe4100a38d0 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe410122a50 con 0x7fe4100a38d0 2024-09-16T10:31:56.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe3f8004b20 con 0x7fe4100a38d0 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): dump: 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336567318 0 0) 0x7fe4101224d0 con 0x7fe41006c670 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 10 monclient(hunting): my global_id is 4748 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f0003520 con 0x7fe41006c670 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2234120863 0 0) 0x7fe410122a50 con 0x7fe4100a38d0 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.838+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 >> v1:172.21.15.152:6791/0 conn(0x7fe41006c670 legacy=0x7fe410132890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 >> v1:172.21.15.152:6790/0 conn(0x7fe4100a43c0 legacy=0x7fe4101d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 1 monclient: found mon.b 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:56.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41009f060 con 0x7fe4100a38d0 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 5 monclient: authenticate success, global_id 4655 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 10 monclient: _renew_subs 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41009eca0 con 0x7fe4100a38d0 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 10 monclient: _renew_subs 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe410120c40 con 0x7fe4100a38d0 2024-09-16T10:31:56.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3f8005210 con 0x7fe4100a38d0 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe3f80056c0 con 0x7fe4100a38d0 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe40dffb640 4 set_mon_vals no callback set 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 10 monclient: dump: 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:56.861 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fe3f8010720 con 0x7fe4100a38d0 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.94:6790/0 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.839+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f4005040 con 0x7fe4100a38d0 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.840+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7fe3f8046260 con 0x7fe4100a38d0 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.840+0000 7fe40effd640 10 monclient: get_auth_request con 0x7fe3f0038f00 auth_method 0 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.844+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fe3f80157d0 con 0x7fe4100a38d0 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.844+0000 7fe4077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:56.844+0000 7fe4077fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:57.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.010+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3f4000b90 con 0x7fe3f0038f00 2024-09-16T10:31:57.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.011+0000 7fe4077fe640 1 -- 172.21.15.94:0/695162154 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (unknown 3782719169 0 0) 0x7fe3f4000b90 con 0x7fe3f0038f00 2024-09-16T10:31:57.013 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T10:31:57.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.013+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 >> v1:172.21.15.152:6800/2686360781 conn(0x7fe3f0038f00 legacy=0x7fe3f003b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.013+0000 7fe4152c9640 10 monclient: shutdown 2024-09-16T10:31:57.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.013+0000 7fe4152c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:57.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.013+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 >> v1:172.21.15.94:6790/0 conn(0x7fe4100a38d0 legacy=0x7fe4101cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.013+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 shutdown_connections 2024-09-16T10:31:57.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.013+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 >> 172.21.15.94:0/695162154 conn(0x7fe4100a59d0 msgr2=0x7fe410135920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:57.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.013+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 shutdown_connections 2024-09-16T10:31:57.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.013+0000 7fe4152c9640 1 -- 172.21.15.94:0/695162154 wait complete. 2024-09-16T10:31:57.025 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T10:31:57.026 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T10:31:57.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.125+0000 7f9ae1b9d640 10 monclient: get_monmap_and_config 2024-09-16T10:31:57.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.125+0000 7f9ae1b9d640 10 monclient: build_initial_monmap 2024-09-16T10:31:57.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.125+0000 7f9ae1b9d640 10 monclient: monmap: 2024-09-16T10:31:57.127 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:57.127 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.127 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:57.126422+0000 2024-09-16T10:31:57.127 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:57.126422+0000 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:57.128 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 1 Processor -- start 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 1 -- start start 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 10 monclient: init 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 10 monclient: _add_conns ranks=[4,0,7,6,5,8,3,2,1] 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f9adc0a5dd0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f9adc136b50 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 10 monclient(hunting): picked mon.noname-i con 0x7f9adc13c9d0 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc04f100 con 0x7f9adc136b50 2024-09-16T10:31:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc0a46f0 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc0a61b0 con 0x7f9adc13c9d0 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae1b9d640 10 monclient(hunting): authenticate will time out at 884.466860s 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae0b9b640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f9adc0a5dd0 0x7f9adc136440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:36228/0 (socket says 172.21.15.94:36228) 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9ae0b9b640 1 -- 172.21.15.94:0/675607905 learned_addr learned my addr 172.21.15.94:0/675607905 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.126+0000 7f9adbfff640 1 --1- 172.21.15.94:0/675607905 >> v1:172.21.15.94:6789/0 conn(0x7f9adc136b50 0x7f9adc13b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57742/0 (socket says 172.21.15.94:57742) 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9ae0b9b640 1 -- 172.21.15.94:0/675607905 >> v1:172.21.15.94:6791/0 conn(0x7f9adc0a5dd0 legacy=0x7f9adc136440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adbfff640 1 -- 172.21.15.94:0/675607905 >> v1:172.21.15.94:6789/0 conn(0x7f9adc136b50 legacy=0x7f9adc13b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957520700 0 0) 0x7f9adc0a46f0 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient(hunting): my global_id is 4738 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9acc003520 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093586295 0 0) 0x7f9adc04f100 con 0x7f9adc136b50 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient(hunting): my global_id is 4644 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9ae139c640 1 -- 172.21.15.94:0/675607905 >> v1:172.21.15.152:6792/0 conn(0x7f9adc13c9d0 legacy=0x7f9adc13edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc0a46f0 con 0x7f9adc136b50 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459454146 0 0) 0x7f9adc0a46f0 con 0x7f9adc136b50 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 >> v1:172.21.15.152:6792/0 conn(0x7f9adc13c9d0 legacy=0x7f9adc13edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 >> v1:172.21.15.94:6791/0 conn(0x7f9adc0a5dd0 legacy=0x7f9adc136440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 monclient: found mon.noname-a 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc09ab90 con 0x7f9adc136b50 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9ae1b9d640 5 monclient: authenticate success, global_id 4644 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9ac8001b20 con 0x7f9adc136b50 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: dump: 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.138 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9ae1b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ac80047e0 con 0x7f9adc136b50 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 1 -- 172.21.15.94:0/675607905 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9ac8004c90 con 0x7f9adc136b50 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9ae1b9d640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9ae1b9d640 4 set_mon_vals no callback set 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:31:57.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.129+0000 7f9adb7fe640 10 monclient: dump: 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.130+0000 7f9ae1b9d640 10 monclient: shutdown 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.130+0000 7f9ae1b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.130+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/675607905 >> v1:172.21.15.94:6789/0 conn(0x7f9adc136b50 legacy=0x7f9adc13b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.130+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/675607905 shutdown_connections 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.130+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/675607905 >> 172.21.15.94:0/675607905 conn(0x7f9adc0a91f0 msgr2=0x7f9adc0a56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.130+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/675607905 shutdown_connections 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.130+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/675607905 wait complete. 2024-09-16T10:31:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.130+0000 7f9ae1b9d640 10 monclient: build_initial_monmap 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.130+0000 7f9ae1b9d640 10 monclient: monmap: 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 1 Processor -- start 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 1 -- start start 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 10 monclient: init 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 10 monclient: _add_conns ranks=[4,2,7,3,5,6,1,0,8] 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f9adc0a5dd0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f9adc136b50 addr v1:172.21.15.94:6790/0 2024-09-16T10:31:57.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 10 monclient(hunting): picked mon.noname-i con 0x7f9adc13c9d0 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc1355e0 con 0x7f9adc136b50 2024-09-16T10:31:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc13c480 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc09dd60 con 0x7f9adc13c9d0 2024-09-16T10:31:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae1b9d640 10 monclient(hunting): authenticate will time out at 884.471879s 2024-09-16T10:31:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae0b9b640 1 --1- >> v1:172.21.15.94:6791/0 conn(0x7f9adc0a5dd0 0x7f9adc1cab50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6791/0 says I am v1:172.21.15.94:36244/0 (socket says 172.21.15.94:36244) 2024-09-16T10:31:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.131+0000 7f9ae0b9b640 1 -- 172.21.15.94:0/4202646523 learned_addr learned my addr 172.21.15.94:0/4202646523 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ae0b9b640 1 -- 172.21.15.94:0/4202646523 >> v1:172.21.15.94:6791/0 conn(0x7f9adc0a5dd0 legacy=0x7f9adc1cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9adbfff640 1 -- 172.21.15.94:0/4202646523 >> v1:172.21.15.94:6790/0 conn(0x7f9adc136b50 legacy=0x7f9adc1cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707352958 0 0) 0x7f9adc13c480 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): my global_id is 4747 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac0003520 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743910633 0 0) 0x7f9adc1355e0 con 0x7f9adc136b50 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): my global_id is 4664 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc13c480 con 0x7f9adc136b50 2024-09-16T10:31:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ae139c640 1 -- 172.21.15.94:0/4202646523 >> v1:172.21.15.152:6792/0 conn(0x7f9adc13c9d0 legacy=0x7f9adc1d2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1967008766 0 0) 0x7f9ac0003520 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9adc1355e0 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9ad0001320 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): dump: 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2615434045 0 0) 0x7f9adc13c480 con 0x7f9adc136b50 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ac0003520 con 0x7f9adc136b50 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9ac8001ae0 con 0x7f9adc136b50 2024-09-16T10:31:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): dump: 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.151 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587141577 0 0) 0x7f9adc09dd60 con 0x7f9adc13c9d0 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient(hunting): my global_id is 4759 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc13c480 con 0x7f9adc13c9d0 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.4 v1:172.21.15.94:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1952903981 0 0) 0x7f9adc1355e0 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 >> v1:172.21.15.152:6792/0 conn(0x7f9adc13c9d0 legacy=0x7f9adc1d2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 >> v1:172.21.15.94:6790/0 conn(0x7f9adc136b50 legacy=0x7f9adc1cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient: _finish_hunting 0 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 monclient: found mon.c 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.94:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc1cb970 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient: _finish_auth 0 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient: _check_auth_tickets 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ae1b9d640 5 monclient: authenticate success, global_id 4747 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ae1b9d640 10 monclient: _renew_subs 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ae1b9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.94:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc09da80 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ae1b9d640 10 monclient: _renew_subs 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ae1b9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.94:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc13b770 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.4 v1:172.21.15.94:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ad0004dd0 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.4 v1:172.21.15.94:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9ad0005280 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9adaffd640 4 set_mon_vals no callback set 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.134+0000 7f9ad97fa640 10 monclient: dump: 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.154 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.155 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.135+0000 7f9ae1b9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6791/0 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.135+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.94:6791/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9abc000f10 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.135+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.4 v1:172.21.15.94:6791/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f9ad00102e0 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.135+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.4 v1:172.21.15.94:6791/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f9ad0045f50 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.135+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.152:6806/3657457384 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ac0043eb0 con 0x7f9ac0040240 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.135+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== mon.4 v1:172.21.15.94:6791/0 9 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (unknown 3265615600 0 0) 0x7f9ad0046340 con 0x7f9adc0a5dd0 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.135+0000 7f9ad97fa640 10 monclient: handle_get_version_reply finishing 1 version 22 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.135+0000 7f9adbfff640 10 monclient: get_auth_request con 0x7f9ac00397c0 auth_method 0 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.136+0000 7f9ae139c640 10 monclient: get_auth_request con 0x7f9ac0040240 auth_method 0 2024-09-16T10:31:57.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.141+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== osd.5 v1:172.21.15.152:6806/3657457384 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f9ac0043eb0 con 0x7f9ac0040240 2024-09-16T10:31:57.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.153+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 --> v1:172.21.15.152:6806/3657457384 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9abc002b50 con 0x7f9ac0040240 2024-09-16T10:31:57.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.153+0000 7f9ad97fa640 1 -- 172.21.15.94:0/4202646523 <== osd.5 v1:172.21.15.152:6806/3657457384 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f9abc002b50 con 0x7f9ac0040240 2024-09-16T10:31:57.157 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-16T10:31:57.157 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T10:31:57.157 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-16T10:31:57.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.153+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 >> v1:172.21.15.152:6806/3657457384 conn(0x7f9ac0040240 legacy=0x7f9ac0042660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.153+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 >> v1:172.21.15.152:6800/2686360781 conn(0x7f9ac00397c0 legacy=0x7f9ac003bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.153+0000 7f9ae1b9d640 10 monclient: shutdown 2024-09-16T10:31:57.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.153+0000 7f9ae1b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:57.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.154+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 >> v1:172.21.15.94:6791/0 conn(0x7f9adc0a5dd0 legacy=0x7f9adc1cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.154+0000 7f9ae139c640 1 -- 172.21.15.94:0/4202646523 reap_dead start 2024-09-16T10:31:57.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.154+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 shutdown_connections 2024-09-16T10:31:57.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.154+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 >> 172.21.15.94:0/4202646523 conn(0x7f9adc0a91f0 msgr2=0x7f9adc0a6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:57.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.154+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 shutdown_connections 2024-09-16T10:31:57.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.154+0000 7f9ae1b9d640 1 -- 172.21.15.94:0/4202646523 wait complete. 2024-09-16T10:31:57.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T10:31:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.260+0000 7f5e12f18640 10 monclient: get_monmap_and_config 2024-09-16T10:31:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.260+0000 7f5e12f18640 10 monclient: build_initial_monmap 2024-09-16T10:31:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.260+0000 7f5e12f18640 10 monclient: monmap: 2024-09-16T10:31:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:57.261664+0000 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:57.261664+0000 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:57.264 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:57.264 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:57.264 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:57.264 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:57.264 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.261+0000 7f5e12f18640 1 Processor -- start 2024-09-16T10:31:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.261+0000 7f5e12f18640 1 -- start start 2024-09-16T10:31:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.261+0000 7f5e12f18640 10 monclient: init 2024-09-16T10:31:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 10 monclient: _add_conns ranks=[7,3,6,1,4,0,8,2,5] 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 10 monclient(hunting): picked mon.noname-i con 0x7f5e0c09bb70 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 10 monclient(hunting): picked mon.noname-g con 0x7f5e0c0a0ba0 addr v1:172.21.15.152:6790/0 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 10 monclient(hunting): picked mon.noname-d con 0x7f5e0c1395a0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c061ae0 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c09dc50 con 0x7f5e0c1395a0 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c09df30 con 0x7f5e0c09bb70 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e12f18640 10 monclient(hunting): authenticate will time out at 884.602242s 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e114b5640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f5e0c1395a0 0x7f5e0c13b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:33842/0 (socket says 172.21.15.94:33842) 2024-09-16T10:31:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.262+0000 7f5e114b5640 1 -- 172.21.15.94:0/4090028844 learned_addr learned my addr 172.21.15.94:0/4090028844 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:57.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e114b5640 1 -- 172.21.15.94:0/4090028844 >> v1:172.21.15.94:6792/0 conn(0x7f5e0c1395a0 legacy=0x7f5e0c13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e10cb4640 1 -- 172.21.15.94:0/4090028844 >> v1:172.21.15.152:6792/0 conn(0x7f5e0c09bb70 legacy=0x7f5e0c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e03fff640 1 -- 172.21.15.94:0/4090028844 >> v1:172.21.15.152:6790/0 conn(0x7f5e0c0a0ba0 legacy=0x7f5e0c09c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035514459 0 0) 0x7f5e0c09dc50 con 0x7f5e0c1395a0 2024-09-16T10:31:57.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): my global_id is 94299 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df4003520 con 0x7f5e0c1395a0 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 <== mon.7 v1:172.21.15.152:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316079628 0 0) 0x7f5e0c09df30 con 0x7f5e0c09bb70 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): my global_id is 4768 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 --> v1:172.21.15.152:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c09dc50 con 0x7f5e0c09bb70 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521281600 0 0) 0x7f5e0c061ae0 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient(hunting): my global_id is 4944 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c09df30 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2700580302 0 0) 0x7f5df4003520 con 0x7f5e0c1395a0 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 >> v1:172.21.15.152:6792/0 conn(0x7f5e0c09bb70 legacy=0x7f5e0c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 >> v1:172.21.15.152:6790/0 conn(0x7f5e0c0a0ba0 legacy=0x7f5e0c09c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 monclient: found mon.noname-d 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c09eff0 con 0x7f5e0c1395a0 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:57.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5e04001ea0 con 0x7f5e0c1395a0 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e037fe640 10 monclient: dump: 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.273 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.264+0000 7f5e12f18640 5 monclient: authenticate success, global_id 94299 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e04004c20 con 0x7f5e0c1395a0 2024-09-16T10:31:57.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e037fe640 1 -- 172.21.15.94:0/4090028844 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5e040050d0 con 0x7f5e0c1395a0 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 4 set_mon_vals no callback set 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e037fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e037fe640 10 monclient: dump: 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.275 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 10 monclient: shutdown 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 1 -- 172.21.15.94:0/4090028844 >> v1:172.21.15.94:6792/0 conn(0x7f5e0c1395a0 legacy=0x7f5e0c13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 1 -- 172.21.15.94:0/4090028844 shutdown_connections 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 1 -- 172.21.15.94:0/4090028844 >> 172.21.15.94:0/4090028844 conn(0x7f5e0c0a1680 msgr2=0x7f5e0c0a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 1 -- 172.21.15.94:0/4090028844 shutdown_connections 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.265+0000 7f5e12f18640 1 -- 172.21.15.94:0/4090028844 wait complete. 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.266+0000 7f5e12f18640 10 monclient: build_initial_monmap 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.266+0000 7f5e12f18640 10 monclient: monmap: 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:57.277 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.266+0000 7f5e12f18640 1 Processor -- start 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.266+0000 7f5e12f18640 1 -- start start 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.266+0000 7f5e12f18640 10 monclient: init 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 10 monclient: _add_conns ranks=[7,6,4,8,2,1,3,0,5] 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 10 monclient(hunting): picked mon.noname-i con 0x7f5e0c09bb70 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 10 monclient(hunting): picked mon.noname-d con 0x7f5e0c0a0ba0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0c1395a0 addr v1:172.21.15.94:6791/0 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c070a50 con 0x7f5e0c1395a0 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c0a2fe0 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c0704b0 con 0x7f5e0c09bb70 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e12f18640 10 monclient(hunting): authenticate will time out at 884.607256s 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e03fff640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f5e0c0a0ba0 0x7f5e0c1cdb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:33856/0 (socket says 172.21.15.94:33856) 2024-09-16T10:31:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.267+0000 7f5e03fff640 1 -- 172.21.15.94:0/2460889028 learned_addr learned my addr 172.21.15.94:0/2460889028 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e03fff640 1 -- 172.21.15.94:0/2460889028 >> v1:172.21.15.94:6792/0 conn(0x7f5e0c0a0ba0 legacy=0x7f5e0c1cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e114b5640 1 -- 172.21.15.94:0/2460889028 >> v1:172.21.15.94:6791/0 conn(0x7f5e0c1395a0 legacy=0x7f5e0c1d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165020315 0 0) 0x7f5e0c0a2fe0 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): my global_id is 94308 2024-09-16T10:31:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df8003520 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194716667 0 0) 0x7f5e0c09eff0 con 0x7f5e0c1395a0 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): my global_id is 4756 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c0a2fe0 con 0x7f5e0c1395a0 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e10cb4640 1 -- 172.21.15.94:0/2460889028 >> v1:172.21.15.152:6792/0 conn(0x7f5e0c09bb70 legacy=0x7f5e0c138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2765980595 0 0) 0x7f5df8003520 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e0c09eff0 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5dfc001d50 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:57.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): dump: 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.285 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1412226759 0 0) 0x7f5e0c0a2fe0 con 0x7f5e0c1395a0 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5df8003520 con 0x7f5e0c1395a0 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5e04001e60 con 0x7f5e0c1395a0 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.269+0000 7f5e017fa640 10 monclient(hunting): dump: 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.286 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.287 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3941637199 0 0) 0x7f5e0c09eff0 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 >> v1:172.21.15.152:6792/0 conn(0x7f5e0c09bb70 legacy=0x7f5e0c138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 >> v1:172.21.15.94:6791/0 conn(0x7f5e0c1395a0 legacy=0x7f5e0c1d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 10 monclient: _finish_hunting 0 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 1 monclient: found mon.d 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c125120 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 10 monclient: _finish_auth 0 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 10 monclient: _check_auth_tickets 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e12f18640 5 monclient: authenticate success, global_id 94308 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e12f18640 10 monclient: _renew_subs 2024-09-16T10:31:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e12f18640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c124880 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e12f18640 10 monclient: _renew_subs 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e12f18640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c123b20 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dfc003b00 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f5dfc003fb0 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e02ffd640 4 set_mon_vals no callback set 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.270+0000 7f5e017fa640 10 monclient: dump: 2024-09-16T10:31:57.289 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.290 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.271+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f5dfc004190 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.271+0000 7f5e12f18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.271+0000 7f5e12f18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.271+0000 7f5e12f18640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.271+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec005040 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.271+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f5dfc045f70 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.271+0000 7f5e10cb4640 10 monclient: get_auth_request con 0x7f5df8039860 auth_method 0 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.275+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f5dfc004440 con 0x7f5e0c0a0ba0 2024-09-16T10:31:57.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.275+0000 7f5e017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:57.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.275+0000 7f5e017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:57.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.434+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f5dec000b90 con 0x7f5df8039860 2024-09-16T10:31:57.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.435+0000 7f5e017fa640 1 -- 172.21.15.94:0/2460889028 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (unknown 2264484800 0 0) 0x7f5dec000b90 con 0x7f5df8039860 2024-09-16T10:31:57.437 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T10:31:57.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.437+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 >> v1:172.21.15.152:6800/2686360781 conn(0x7f5df8039860 legacy=0x7f5df803bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.437+0000 7f5e12f18640 10 monclient: shutdown 2024-09-16T10:31:57.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.437+0000 7f5e12f18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:57.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.437+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 >> v1:172.21.15.94:6792/0 conn(0x7f5e0c0a0ba0 legacy=0x7f5e0c1cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.437+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 shutdown_connections 2024-09-16T10:31:57.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.437+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 >> 172.21.15.94:0/2460889028 conn(0x7f5e0c0a1680 msgr2=0x7f5e0c09aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:57.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.437+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 shutdown_connections 2024-09-16T10:31:57.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.437+0000 7f5e12f18640 1 -- 172.21.15.94:0/2460889028 wait complete. 2024-09-16T10:31:57.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:57.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.544+0000 7fe805325640 10 monclient: get_monmap_and_config 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.544+0000 7fe805325640 10 monclient: build_initial_monmap 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.544+0000 7fe805325640 10 monclient: monmap: 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:31:57.545165+0000 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:31:57.545165+0000 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:57.547 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:57.547 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:57.547 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:57.547 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:57.547 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:57.547 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:57.547 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 1 Processor -- start 2024-09-16T10:31:57.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 1 -- start start 2024-09-16T10:31:57.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 10 monclient: init 2024-09-16T10:31:57.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:57.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 10 monclient: _add_conns ranks=[7,0,6,8,3,4,2,1,5] 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 10 monclient(hunting): picked mon.noname-i con 0x7fe8000a3860 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 10 monclient(hunting): picked mon.noname-a con 0x7fe80006c1d0 addr v1:172.21.15.94:6789/0 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 10 monclient(hunting): picked mon.noname-d con 0x7fe8000a4350 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe800061ae0 con 0x7fe80006c1d0 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8000a0600 con 0x7fe8000a4350 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8000a0b30 con 0x7fe8000a3860 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe805325640 10 monclient(hunting): authenticate will time out at 884.885617s 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe7ff7fe640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fe8000a4350 0x7fe80013ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:33868/0 (socket says 172.21.15.94:33868) 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe7ff7fe640 1 -- 172.21.15.94:0/3198170495 learned_addr learned my addr 172.21.15.94:0/3198170495 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:57.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.545+0000 7fe7f7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe80006c1d0 0x7fe80013b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57752/0 (socket says 172.21.15.94:57752) 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.547+0000 7fe7f7fff640 1 -- 172.21.15.94:0/3198170495 >> v1:172.21.15.94:6789/0 conn(0x7fe80006c1d0 legacy=0x7fe80013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.547+0000 7fe7ff7fe640 1 -- 172.21.15.94:0/3198170495 >> v1:172.21.15.94:6792/0 conn(0x7fe8000a4350 legacy=0x7fe80013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003193524 0 0) 0x7fe8000a0600 con 0x7fe8000a4350 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient(hunting): my global_id is 94317 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7ec003520 con 0x7fe8000a4350 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863723500 0 0) 0x7fe800061ae0 con 0x7fe80006c1d0 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient(hunting): my global_id is 4653 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8000a0600 con 0x7fe80006c1d0 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7feffd640 1 -- 172.21.15.94:0/3198170495 >> v1:172.21.15.152:6792/0 conn(0x7fe8000a3860 legacy=0x7fe8000a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3861512062 0 0) 0x7fe7ec003520 con 0x7fe8000a4350 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 >> v1:172.21.15.152:6792/0 conn(0x7fe8000a3860 legacy=0x7fe8000a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 >> v1:172.21.15.94:6789/0 conn(0x7fe80006c1d0 legacy=0x7fe80013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: _finish_hunting 0 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 monclient: found mon.noname-d 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8000a0e10 con 0x7fe8000a4350 2024-09-16T10:31:57.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: _finish_auth 0 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: _check_auth_tickets 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe7f8001de0 con 0x7fe8000a4350 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: dump: 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.555 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe805325640 5 monclient: authenticate success, global_id 94317 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe805325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:31:57.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 <== mon.6 v1:172.21.15.94:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7f80020f0 con 0x7fe8000a4350 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe805325640 10 monclient: get_monmap_and_config success 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 1 -- 172.21.15.94:0/3198170495 <== mon.6 v1:172.21.15.94:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe7f8004f60 con 0x7fe8000a4350 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe805325640 4 set_mon_vals no callback set 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe7fe7fc640 10 monclient: dump: 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.557 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe805325640 10 monclient: shutdown 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe805325640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.548+0000 7fe805325640 1 -- 172.21.15.94:0/3198170495 >> v1:172.21.15.94:6792/0 conn(0x7fe8000a4350 legacy=0x7fe80013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.549+0000 7fe805325640 1 -- 172.21.15.94:0/3198170495 shutdown_connections 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.549+0000 7fe805325640 1 -- 172.21.15.94:0/3198170495 >> 172.21.15.94:0/3198170495 conn(0x7fe8000a5960 msgr2=0x7fe8000a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.549+0000 7fe805325640 1 -- 172.21.15.94:0/3198170495 shutdown_connections 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.549+0000 7fe805325640 1 -- 172.21.15.94:0/3198170495 wait complete. 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.549+0000 7fe805325640 10 monclient: build_initial_monmap 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.549+0000 7fe805325640 10 monclient: monmap: 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:31:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.549+0000 7fe805325640 1 Processor -- start 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 1 -- start start 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 10 monclient: init 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 10 monclient: _reopen_session rank -1 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 10 monclient: _add_conns ranks=[6,1,7,4,2,5,0,3,8] 2024-09-16T10:31:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 10 monclient(hunting): picked mon.noname-d con 0x7fe80006c1d0 addr v1:172.21.15.94:6792/0 2024-09-16T10:31:57.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 10 monclient(hunting): picked mon.noname-f con 0x7fe8000a3860 addr v1:172.21.15.152:6789/0 2024-09-16T10:31:57.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 10 monclient(hunting): picked mon.noname-i con 0x7fe8000a4350 addr v1:172.21.15.152:6792/0 2024-09-16T10:31:57.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8001a9620 con 0x7fe8000a3860 2024-09-16T10:31:57.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80009f160 con 0x7fe80006c1d0 2024-09-16T10:31:57.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80009f6b0 con 0x7fe8000a4350 2024-09-16T10:31:57.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 10 monclient(hunting): _renew_subs 2024-09-16T10:31:57.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe7feffd640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7fe80006c1d0 0x7fe8001bd630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:33872/0 (socket says 172.21.15.94:33872) 2024-09-16T10:31:57.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe805325640 10 monclient(hunting): authenticate will time out at 884.890520s 2024-09-16T10:31:57.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.550+0000 7fe7feffd640 1 -- 172.21.15.94:0/2142040552 learned_addr learned my addr 172.21.15.94:0/2142040552 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:31:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7feffd640 1 -- 172.21.15.94:0/2142040552 >> v1:172.21.15.94:6792/0 conn(0x7fe80006c1d0 legacy=0x7fe8001bd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930446123 0 0) 0x7fe80009f160 con 0x7fe80006c1d0 2024-09-16T10:31:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 10 monclient(hunting): _negotiate 2024-09-16T10:31:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:31:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:31:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 10 monclient(hunting): my global_id is 94326 2024-09-16T10:31:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e0003520 con 0x7fe80006c1d0 2024-09-16T10:31:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 291258586 0 0) 0x7fe7e0003520 con 0x7fe80006c1d0 2024-09-16T10:31:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe80009f160 con 0x7fe80006c1d0 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe7f0001d30 con 0x7fe80006c1d0 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7ff7fe640 1 -- 172.21.15.94:0/2142040552 >> v1:172.21.15.152:6792/0 conn(0x7fe8000a4350 legacy=0x7fe8001d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f7fff640 1 -- 172.21.15.94:0/2142040552 >> v1:172.21.15.152:6789/0 conn(0x7fe8000a3860 legacy=0x7fe8001cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.552+0000 7fe7f77fe640 10 monclient(hunting): dump: 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 475752777 0 0) 0x7fe80009f160 con 0x7fe80006c1d0 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 >> v1:172.21.15.152:6792/0 conn(0x7fe8000a4350 legacy=0x7fe8001d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 >> v1:172.21.15.152:6789/0 conn(0x7fe8000a3860 legacy=0x7fe8001cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 10 monclient: _finish_hunting 0 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 1 monclient: found mon.d 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80006bae0 con 0x7fe80006c1d0 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 10 monclient: _finish_auth 0 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 10 monclient: _check_auth_tickets 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe805325640 5 monclient: authenticate success, global_id 94326 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe805325640 10 monclient: _renew_subs 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe805325640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8001aad10 con 0x7fe80006c1d0 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe805325640 10 monclient: _renew_subs 2024-09-16T10:31:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe805325640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8001a8c20 con 0x7fe80006c1d0 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7f0003bf0 con 0x7fe80006c1d0 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fe7f00040a0 con 0x7fe80006c1d0 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7f77fe640 10 monclient: dump: 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:31:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.553+0000 7fe7fdffb640 4 set_mon_vals no callback set 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.554+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fe7f0004200 con 0x7fe80006c1d0 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.554+0000 7fe805325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.554+0000 7fe805325640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:57.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.554+0000 7fe805325640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:31:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.554+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8000008d0 con 0x7fe80006c1d0 2024-09-16T10:31:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.554+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7fe7f0046010 con 0x7fe80006c1d0 2024-09-16T10:31:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.554+0000 7fe7f7fff640 10 monclient: get_auth_request con 0x7fe7e00390e0 auth_method 0 2024-09-16T10:31:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.558+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fe7f0015520 con 0x7fe80006c1d0 2024-09-16T10:31:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.558+0000 7fe7f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:31:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.558+0000 7fe7f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:31:57.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.718+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe8001aa750 con 0x7fe7e00390e0 2024-09-16T10:31:57.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.720+0000 7fe7f77fe640 1 -- 172.21.15.94:0/2142040552 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53975 (unknown 2965378022 0 4258498823) 0x7fe8001aa750 con 0x7fe7e00390e0 2024-09-16T10:31:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:31:57.722 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-09-16T10:31:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.722+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 >> v1:172.21.15.152:6800/2686360781 conn(0x7fe7e00390e0 legacy=0x7fe7e003b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.722+0000 7fe805325640 10 monclient: shutdown 2024-09-16T10:31:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.722+0000 7fe805325640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:31:57.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.722+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 >> v1:172.21.15.94:6792/0 conn(0x7fe80006c1d0 legacy=0x7fe8001bd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:31:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.722+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 shutdown_connections 2024-09-16T10:31:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.722+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 >> 172.21.15.94:0/2142040552 conn(0x7fe8000a5960 msgr2=0x7fe8001be3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:31:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.722+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 shutdown_connections 2024-09-16T10:31:57.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:31:57.722+0000 7fe805325640 1 -- 172.21.15.94:0/2142040552 wait complete. 2024-09-16T10:31:57.734 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-16T10:31:57.032310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176624,"kb_used_data":2200,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054416,"statfs":{"total":579820584960,"available":579639721984,"internally_reserved":0,"allocated":2252800,"data_stored":1470364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002235"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.016774+0000","last_change":"2024-09-16T10:29:32.967697+0000","last_active":"2024-09-16T10:31:42.016774+0000","last_peered":"2024-09-16T10:31:42.016774+0000","last_clean":"2024-09-16T10:31:42.016774+0000","last_became_active":"2024-09-16T10:29:30.052739+0000","last_became_peered":"2024-09-16T10:29:30.052739+0000","last_unstale":"2024-09-16T10:31:42.016774+0000","last_undegraded":"2024-09-16T10:31:42.016774+0000","last_fullsized":"2024-09-16T10:31:42.016774+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035088200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:56.809577+0000","last_change":"2024-09-16T10:31:56.809673+0000","last_active":"2024-09-16T10:31:56.809577+0000","last_peered":"2024-09-16T10:31:56.809577+0000","last_clean":"2024-09-16T10:31:56.809577+0000","last_became_active":"2024-09-16T10:29:30.043984+0000","last_became_peered":"2024-09-16T10:29:30.043984+0000","last_unstale":"2024-09-16T10:31:56.809577+0000","last_undegraded":"2024-09-16T10:31:56.809577+0000","last_fullsized":"2024-09-16T10:31:56.809577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:56.809560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:31:56.809560+0000","last_clean_scrub_stamp":"2024-09-16T10:31:56.809560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:48:37.352420+0000","scrub_duration":0.0016066450000000001,"objects_trimmed":0,"snaptrim_duration":0.00028018599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:37.016421+0000","last_change":"2024-09-16T10:29:32.967858+0000","last_active":"2024-09-16T10:31:37.016421+0000","last_peered":"2024-09-16T10:31:37.016421+0000","last_clean":"2024-09-16T10:31:37.016421+0000","last_became_active":"2024-09-16T10:29:30.043883+0000","last_became_peered":"2024-09-16T10:29:30.043883+0000","last_unstale":"2024-09-16T10:31:37.016421+0000","last_undegraded":"2024-09-16T10:31:37.016421+0000","last_fullsized":"2024-09-16T10:31:37.016421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048663500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:37.016702+0000","last_change":"2024-09-16T10:29:32.789216+0000","last_active":"2024-09-16T10:31:37.016702+0000","last_peered":"2024-09-16T10:31:37.016702+0000","last_clean":"2024-09-16T10:31:37.016702+0000","last_became_active":"2024-09-16T10:29:30.039372+0000","last_became_peered":"2024-09-16T10:29:30.039372+0000","last_unstale":"2024-09-16T10:31:37.016702+0000","last_undegraded":"2024-09-16T10:31:37.016702+0000","last_fullsized":"2024-09-16T10:31:37.016702+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027726900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.412098+0000","last_change":"2024-09-16T10:29:32.794417+0000","last_active":"2024-09-16T10:30:04.412098+0000","last_peered":"2024-09-16T10:30:04.412098+0000","last_clean":"2024-09-16T10:30:04.412098+0000","last_became_active":"2024-09-16T10:29:30.039308+0000","last_became_peered":"2024-09-16T10:29:30.039308+0000","last_unstale":"2024-09-16T10:30:04.412098+0000","last_undegraded":"2024-09-16T10:30:04.412098+0000","last_fullsized":"2024-09-16T10:30:04.412098+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:04.076079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076855000000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.314171+0000","last_change":"2024-09-16T10:29:32.784986+0000","last_active":"2024-09-16T10:30:04.314171+0000","last_peered":"2024-09-16T10:30:04.314171+0000","last_clean":"2024-09-16T10:30:04.314171+0000","last_became_active":"2024-09-16T10:29:30.039431+0000","last_became_peered":"2024-09-16T10:29:30.039431+0000","last_unstale":"2024-09-16T10:30:04.314171+0000","last_undegraded":"2024-09-16T10:30:04.314171+0000","last_fullsized":"2024-09-16T10:30:04.314171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:20.175515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041305900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.015997+0000","last_change":"2024-09-16T10:29:32.967651+0000","last_active":"2024-09-16T10:31:32.015997+0000","last_peered":"2024-09-16T10:31:32.015997+0000","last_clean":"2024-09-16T10:31:32.015997+0000","last_became_active":"2024-09-16T10:29:30.043748+0000","last_became_peered":"2024-09-16T10:29:30.043748+0000","last_unstale":"2024-09-16T10:31:32.015997+0000","last_undegraded":"2024-09-16T10:31:32.015997+0000","last_fullsized":"2024-09-16T10:31:32.015997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349687,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:30:04.411572+0000","last_change":"2024-09-16T10:29:32.794058+0000","last_active":"2024-09-16T10:30:04.411572+0000","last_peered":"2024-09-16T10:30:04.411572+0000","last_clean":"2024-09-16T10:30:04.411572+0000","last_became_active":"2024-09-16T10:29:30.038374+0000","last_became_peered":"2024-09-16T10:29:30.038374+0000","last_unstale":"2024-09-16T10:30:04.411572+0000","last_undegraded":"2024-09-16T10:30:04.411572+0000","last_fullsized":"2024-09-16T10:30:04.411572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:28.997899+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.997899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:17.735950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040381400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.016035+0000","last_change":"2024-09-16T10:29:29.005269+0000","last_active":"2024-09-16T10:31:32.016035+0000","last_peered":"2024-09-16T10:31:32.016035+0000","last_clean":"2024-09-16T10:31:32.016035+0000","last_became_active":"2024-09-16T10:29:29.004894+0000","last_became_peered":"2024-09-16T10:29:29.004894+0000","last_unstale":"2024-09-16T10:31:32.016035+0000","last_undegraded":"2024-09-16T10:31:32.016035+0000","last_fullsized":"2024-09-16T10:31:32.016035+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:27.994140+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.994140+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640289,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.502,"15min":0.49199999999999999},"min":{"1min":0.39400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66300000000000003,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.45100000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.496},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.47599999999999998,"15min":0.46700000000000003},"min":{"1min":0.40799999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.433}]},{"osd":2,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.35599999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.379},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.503,"15min":0.498},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36599999999999999,"15min":0.36499999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.51000000000000001,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.38800000000000001},"min":{"1min":0.312,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.32300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.53100000000000003}]}]},{"osd":2,"up_from":12,"seq":51539607583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.308,"15min":0.308},"min":{"1min":0.23200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.39200000000000002,"5min":0.439,"15min":0.439},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.442,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.372}]},{"osd":1,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.34300000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42399999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.34599999999999997,"15min":0.34799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.504,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56200000000000006}]},{"osd":4,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58899999999999997}]},{"osd":5,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.66900000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640288,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.627,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.376,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.50900000000000001,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63700000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.44},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.40100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.495}]},{"osd":2,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.40100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.626,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65300000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.55200000000000005,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41899999999999998,"15min":0.41999999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54200000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.438,"15min":0.437},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.38100000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640289,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.505,"15min":0.495},"min":{"1min":0.41099999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68400000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.42299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.627}]},{"osd":1,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.48099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68000000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.46200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.45300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53500000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.67500000000000004}]},{"osd":4,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.32100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.29399999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.61099999999999999}]},{"osd":5,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.66200000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640288,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.309},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.375,"15min":0.37},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54900000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34399999999999997},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.497,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.26800000000000002}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.41099999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78400000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.621},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54800000000000004}]}]},{"osd":0,"up_from":11,"seq":47244640288,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.32700000000000001},"min":{"1min":0.26800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.44600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.375,"15min":0.379},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.41999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.221}]},{"osd":2,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.26500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.38700000000000001,"15min":0.38900000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.48899999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.24299999999999999}]},{"osd":3,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.747,"15min":0.78200000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68899999999999995,"5min":5.032,"15min":5.032},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67600000000000005,"5min":4.9240000000000004,"15min":4.9240000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.35899999999999999}]},{"osd":5,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.75700000000000001,"15min":0.79000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.71999999999999997,"5min":4.9800000000000004,"15min":4.9800000000000004},"last":0.5},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.753,"5min":5.0030000000000001,"15min":5.0030000000000001},"last":0.34499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:57.738 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T10:29:28.997899+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=29, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=31, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:31:57.738 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T10:29:28.997899+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=29, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=31, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:31:57.738 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T10:29:28.997899+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=29, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=31, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:31:57.738 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T10:29:28.997899+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=29, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=31, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:31:57.738 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T10:29:28.997899+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=29, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=31, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:31:57.739 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T10:29:28.997899+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=29, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=31, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:31:57.739 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T10:29:28.997899+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=29, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=31, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:31:57.739 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T10:29:27.994140+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=29, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=31, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:31:57.739 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T10:32:17.740 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.847+0000 7f9fa629c640 10 monclient: get_monmap_and_config 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.847+0000 7f9fa629c640 10 monclient: build_initial_monmap 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.847+0000 7f9fa629c640 10 monclient: monmap: 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:32:17.849320+0000 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:32:17.849320+0000 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:32:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:32:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:32:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:32:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:32:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:32:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:32:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:32:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:32:17.851 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 1 Processor -- start 2024-09-16T10:32:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 1 -- start start 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 10 monclient: init 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 10 monclient: _reopen_session rank -1 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 10 monclient: _add_conns ranks=[5,2,1,3,0,6,4,7,8] 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 10 monclient(hunting): picked mon.noname-h con 0x7f9fa00a3840 addr v1:172.21.15.152:6791/0 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa00a17d0 addr v1:172.21.15.94:6790/0 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 10 monclient(hunting): picked mon.noname-f con 0x7f9fa00a4330 addr v1:172.21.15.152:6789/0 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa0061ae0 con 0x7f9fa00a4330 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa00a3280 con 0x7f9fa00a17d0 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa006c7d0 con 0x7f9fa00a3840 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 10 monclient(hunting): _renew_subs 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9fa629c640 10 monclient(hunting): authenticate will time out at 905.189850s 2024-09-16T10:32:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.848+0000 7f9f9f7fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f9fa00a17d0 0x7f9fa013ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:51120/0 (socket says 172.21.15.94:51120) 2024-09-16T10:32:17.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.849+0000 7f9f9f7fe640 1 -- 172.21.15.94:0/3071656351 learned_addr learned my addr 172.21.15.94:0/3071656351 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:32:17.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9f7fe640 1 -- 172.21.15.94:0/3071656351 >> v1:172.21.15.94:6790/0 conn(0x7f9fa00a17d0 legacy=0x7f9fa013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:17.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9ffff640 1 -- 172.21.15.94:0/3071656351 >> v1:172.21.15.152:6791/0 conn(0x7f9fa00a3840 legacy=0x7f9fa00a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:17.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9fa4839640 1 -- 172.21.15.94:0/3071656351 >> v1:172.21.15.152:6789/0 conn(0x7f9fa00a4330 legacy=0x7f9fa013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440888173 0 0) 0x7f9fa00a3280 con 0x7f9fa00a17d0 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): _negotiate 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): my global_id is 4673 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f88003520 con 0x7f9fa00a17d0 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206713854 0 0) 0x7f9fa006c7d0 con 0x7f9fa00a3840 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): _negotiate 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): my global_id is 4757 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa00a3280 con 0x7f9fa00a3840 2024-09-16T10:32:17.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583177794 0 0) 0x7f9fa0061ae0 con 0x7f9fa00a4330 2024-09-16T10:32:17.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): _negotiate 2024-09-16T10:32:17.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:17.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:17.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient(hunting): my global_id is 4672 2024-09-16T10:32:17.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa006c7d0 con 0x7f9fa00a4330 2024-09-16T10:32:17.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150521083 0 0) 0x7f9f88003520 con 0x7f9fa00a17d0 2024-09-16T10:32:17.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 >> v1:172.21.15.152:6791/0 conn(0x7f9fa00a3840 legacy=0x7f9fa00a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:17.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 >> v1:172.21.15.152:6789/0 conn(0x7f9fa00a4330 legacy=0x7f9fa013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:17.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: _finish_hunting 0 2024-09-16T10:32:17.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 monclient: found mon.noname-b 2024-09-16T10:32:17.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa00a0550 con 0x7f9fa00a17d0 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: _finish_auth 0 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: _check_auth_tickets 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9f94001a60 con 0x7f9fa00a17d0 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: dump: 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.861 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9fa629c640 5 monclient: authenticate success, global_id 4673 2024-09-16T10:32:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9fa629c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f94001d70 con 0x7f9fa00a17d0 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 1 -- 172.21.15.94:0/3071656351 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9f94004be0 con 0x7f9fa00a17d0 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9fa629c640 10 monclient: get_monmap_and_config success 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9fa629c640 4 set_mon_vals no callback set 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.851+0000 7f9f9effd640 10 monclient: dump: 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.863 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.852+0000 7f9fa629c640 10 monclient: shutdown 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.852+0000 7f9fa629c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.852+0000 7f9fa629c640 1 -- 172.21.15.94:0/3071656351 >> v1:172.21.15.94:6790/0 conn(0x7f9fa00a17d0 legacy=0x7f9fa013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.852+0000 7f9fa629c640 1 -- 172.21.15.94:0/3071656351 shutdown_connections 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.852+0000 7f9fa629c640 1 -- 172.21.15.94:0/3071656351 >> 172.21.15.94:0/3071656351 conn(0x7f9fa00a97b0 msgr2=0x7f9fa00a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.852+0000 7f9fa629c640 1 -- 172.21.15.94:0/3071656351 shutdown_connections 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.852+0000 7f9fa629c640 1 -- 172.21.15.94:0/3071656351 wait complete. 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.853+0000 7f9fa629c640 10 monclient: build_initial_monmap 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.853+0000 7f9fa629c640 10 monclient: monmap: 2024-09-16T10:32:17.865 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:32:17.866 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.853+0000 7f9fa629c640 1 Processor -- start 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.853+0000 7f9fa629c640 1 -- start start 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.853+0000 7f9fa629c640 10 monclient: init 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.853+0000 7f9fa629c640 10 monclient: _reopen_session rank -1 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.853+0000 7f9fa629c640 10 monclient: _add_conns ranks=[6,4,8,1,2,7,0,3,5] 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9fa629c640 10 monclient(hunting): picked mon.noname-d con 0x7f9fa00a3840 addr v1:172.21.15.94:6792/0 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9fa629c640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa00a4330 addr v1:172.21.15.94:6791/0 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9fa629c640 10 monclient(hunting): picked mon.noname-e con 0x7f9fa0136eb0 addr v1:172.21.15.94:6793/0 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9fa629c640 1 -- --> v1:172.21.15.94:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa0123ba0 con 0x7f9fa00a4330 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9fa629c640 1 -- --> v1:172.21.15.94:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa009f0e0 con 0x7f9fa00a3840 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9fa629c640 1 -- --> v1:172.21.15.94:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa006ba50 con 0x7f9fa0136eb0 2024-09-16T10:32:17.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9fa629c640 10 monclient(hunting): _renew_subs 2024-09-16T10:32:17.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9fa629c640 10 monclient(hunting): authenticate will time out at 905.195084s 2024-09-16T10:32:17.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9f9ffff640 1 --1- >> v1:172.21.15.94:6792/0 conn(0x7f9fa00a3840 0x7f9fa01367a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6792/0 says I am v1:172.21.15.94:35544/0 (socket says 172.21.15.94:35544) 2024-09-16T10:32:17.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.854+0000 7f9f9ffff640 1 -- 172.21.15.94:0/1508631334 learned_addr learned my addr 172.21.15.94:0/1508631334 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:32:17.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9ffff640 1 -- 172.21.15.94:0/1508631334 >> v1:172.21.15.94:6792/0 conn(0x7f9fa00a3840 legacy=0x7f9fa01367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:17.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9fa4839640 1 -- 172.21.15.94:0/1508631334 >> v1:172.21.15.94:6793/0 conn(0x7f9fa0136eb0 legacy=0x7f9fa01d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:17.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9f7fe640 1 -- 172.21.15.94:0/1508631334 >> v1:172.21.15.94:6791/0 conn(0x7f9fa00a4330 legacy=0x7f9fa01cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:17.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.6 v1:172.21.15.94:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498407435 0 0) 0x7f9fa00a3280 con 0x7f9fa00a3840 2024-09-16T10:32:17.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _negotiate 2024-09-16T10:32:17.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): my global_id is 94335 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8c003520 con 0x7f9fa00a3840 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.8 v1:172.21.15.94:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516189273 0 0) 0x7f9fa006ba50 con 0x7f9fa0136eb0 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _negotiate 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): my global_id is 4706 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6793/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa00a3280 con 0x7f9fa0136eb0 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.4 v1:172.21.15.94:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988603269 0 0) 0x7f9fa0123ba0 con 0x7f9fa00a4330 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _negotiate 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:17.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): my global_id is 4765 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa006ba50 con 0x7f9fa00a4330 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.6 v1:172.21.15.94:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036979830 0 0) 0x7f9f8c003520 con 0x7f9fa00a3840 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fa0123ba0 con 0x7f9fa00a3840 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.4 v1:172.21.15.94:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587231780 0 0) 0x7f9fa006ba50 con 0x7f9fa00a4330 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f8c003520 con 0x7f9fa00a4330 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.8 v1:172.21.15.94:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593907472 0 0) 0x7f9fa00a3280 con 0x7f9fa0136eb0 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6793/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fa006ba50 con 0x7f9fa0136eb0 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.6 v1:172.21.15.94:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9f90003610 con 0x7f9fa00a3840 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): dump: 2024-09-16T10:32:17.873 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:17.874 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.4 v1:172.21.15.94:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9f94004b20 con 0x7f9fa00a4330 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): dump: 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:17.875 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.8 v1:172.21.15.94:6793/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9f98001ff0 con 0x7f9fa0136eb0 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 10 monclient(hunting): dump: 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:17.877 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.6 v1:172.21.15.94:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2541678541 0 0) 0x7f9fa0123ba0 con 0x7f9fa00a3840 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 >> v1:172.21.15.94:6793/0 conn(0x7f9fa0136eb0 legacy=0x7f9fa01d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.856+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 >> v1:172.21.15.94:6791/0 conn(0x7f9fa00a4330 legacy=0x7f9fa01cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 10 monclient: _finish_hunting 0 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 1 monclient: found mon.d 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa01238c0 con 0x7f9fa00a3840 2024-09-16T10:32:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 10 monclient: _finish_auth 0 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 10 monclient: _check_auth_tickets 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9fa629c640 5 monclient: authenticate success, global_id 94335 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9fa629c640 10 monclient: _renew_subs 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9fa629c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa0122b80 con 0x7f9fa00a3840 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9fa629c640 10 monclient: _renew_subs 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9fa629c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa01caaa0 con 0x7f9fa00a3840 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.6 v1:172.21.15.94:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f90003be0 con 0x7f9fa00a3840 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.6 v1:172.21.15.94:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7f9f90004090 con 0x7f9fa00a3840 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:17.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 10 monclient: dump: 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:17.880 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9e7fc640 4 set_mon_vals no callback set 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.857+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.6 v1:172.21.15.94:6792/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7f9f90004270 con 0x7f9fa00a3840 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.858+0000 7f9fa629c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.858+0000 7f9fa629c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.858+0000 7f9fa629c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.94:6792/0 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.858+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.94:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa0067530 con 0x7f9fa00a3840 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.858+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.6 v1:172.21.15.94:6792/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7f9f90045e20 con 0x7f9fa00a3840 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.858+0000 7f9f9f7fe640 10 monclient: get_auth_request con 0x7f9f8c039ce0 auth_method 0 2024-09-16T10:32:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.862+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mon.6 v1:172.21.15.94:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7f9f90015330 con 0x7f9fa00a3840 2024-09-16T10:32:17.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.862+0000 7f9f9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:32:17.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:17.862+0000 7f9f9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:32:18.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.024+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 --> v1:172.21.15.152:6800/2686360781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9fa00a0b10 con 0x7f9f8c039ce0 2024-09-16T10:32:18.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.026+0000 7f9f9cff9640 1 -- 172.21.15.94:0/1508631334 <== mgr.4109 v1:172.21.15.152:6800/2686360781 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+54303 (unknown 2965378022 0 200125580) 0x7f9f94001540 con 0x7f9f8c039ce0 2024-09-16T10:32:18.029 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-16T10:32:18.030 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-09-16T10:32:18.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.028+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 >> v1:172.21.15.152:6800/2686360781 conn(0x7f9f8c039ce0 legacy=0x7f9f8c03c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:18.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.028+0000 7f9fa629c640 10 monclient: shutdown 2024-09-16T10:32:18.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.028+0000 7f9fa629c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:32:18.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.028+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 >> v1:172.21.15.94:6792/0 conn(0x7f9fa00a3840 legacy=0x7f9fa01367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:18.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.028+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 shutdown_connections 2024-09-16T10:32:18.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.028+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 >> 172.21.15.94:0/1508631334 conn(0x7f9fa00a97b0 msgr2=0x7f9fa01be750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:32:18.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.028+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 shutdown_connections 2024-09-16T10:32:18.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.028+0000 7f9fa629c640 1 -- 172.21.15.94:0/1508631334 wait complete. 2024-09-16T10:32:18.041 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-16T10:32:17.035355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176588,"kb_used_data":2200,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054452,"statfs":{"total":579820584960,"available":579639758848,"internally_reserved":0,"allocated":2252800,"data_stored":1470364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001873"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.796929+0000","last_change":"2024-09-16T10:32:03.797055+0000","last_active":"2024-09-16T10:32:03.796929+0000","last_peered":"2024-09-16T10:32:03.796929+0000","last_clean":"2024-09-16T10:32:03.796929+0000","last_became_active":"2024-09-16T10:29:30.052739+0000","last_became_peered":"2024-09-16T10:29:30.052739+0000","last_unstale":"2024-09-16T10:32:03.796929+0000","last_undegraded":"2024-09-16T10:32:03.796929+0000","last_fullsized":"2024-09-16T10:32:03.796929+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:03.796906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:32:03.796906+0000","last_clean_scrub_stamp":"2024-09-16T10:32:03.796906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:01.891346+0000","scrub_duration":0.0013895240000000001,"objects_trimmed":0,"snaptrim_duration":0.00035088200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:56.809577+0000","last_change":"2024-09-16T10:31:56.809673+0000","last_active":"2024-09-16T10:31:56.809577+0000","last_peered":"2024-09-16T10:31:56.809577+0000","last_clean":"2024-09-16T10:31:56.809577+0000","last_became_active":"2024-09-16T10:29:30.043984+0000","last_became_peered":"2024-09-16T10:29:30.043984+0000","last_unstale":"2024-09-16T10:31:56.809577+0000","last_undegraded":"2024-09-16T10:31:56.809577+0000","last_fullsized":"2024-09-16T10:31:56.809577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:56.809560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:31:56.809560+0000","last_clean_scrub_stamp":"2024-09-16T10:31:56.809560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:48:37.352420+0000","scrub_duration":0.0016066450000000001,"objects_trimmed":0,"snaptrim_duration":0.00028018599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.797068+0000","last_change":"2024-09-16T10:32:00.797156+0000","last_active":"2024-09-16T10:32:00.797068+0000","last_peered":"2024-09-16T10:32:00.797068+0000","last_clean":"2024-09-16T10:32:00.797068+0000","last_became_active":"2024-09-16T10:29:30.043883+0000","last_became_peered":"2024-09-16T10:29:30.043883+0000","last_unstale":"2024-09-16T10:32:00.797068+0000","last_undegraded":"2024-09-16T10:32:00.797068+0000","last_fullsized":"2024-09-16T10:32:00.797068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:00.797047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:32:00.797047+0000","last_clean_scrub_stamp":"2024-09-16T10:32:00.797047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:10:53.345256+0000","scrub_duration":0.001300837,"objects_trimmed":0,"snaptrim_duration":0.00048663500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.774055+0000","last_change":"2024-09-16T10:31:57.774129+0000","last_active":"2024-09-16T10:31:57.774055+0000","last_peered":"2024-09-16T10:31:57.774055+0000","last_clean":"2024-09-16T10:31:57.774055+0000","last_became_active":"2024-09-16T10:29:30.039372+0000","last_became_peered":"2024-09-16T10:29:30.039372+0000","last_unstale":"2024-09-16T10:31:57.774055+0000","last_undegraded":"2024-09-16T10:31:57.774055+0000","last_fullsized":"2024-09-16T10:31:57.774055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:57.774035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:31:57.774035+0000","last_clean_scrub_stamp":"2024-09-16T10:31:57.774035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:42:47.495756+0000","scrub_duration":0.0010519290000000001,"objects_trimmed":0,"snaptrim_duration":0.00027726900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"22'3","reported_seq":38,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:59.444949+0000","last_change":"2024-09-16T10:31:59.445040+0000","last_active":"2024-09-16T10:31:59.444949+0000","last_peered":"2024-09-16T10:31:59.444949+0000","last_clean":"2024-09-16T10:31:59.444949+0000","last_became_active":"2024-09-16T10:29:30.039308+0000","last_became_peered":"2024-09-16T10:29:30.039308+0000","last_unstale":"2024-09-16T10:31:59.444949+0000","last_undegraded":"2024-09-16T10:31:59.444949+0000","last_fullsized":"2024-09-16T10:31:59.444949+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-09-16T10:31:59.444927+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-09-16T10:31:59.444927+0000","last_clean_scrub_stamp":"2024-09-16T10:31:59.444927+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:15:54.003620+0000","scrub_duration":0.0032606660000000002,"objects_trimmed":0,"snaptrim_duration":0.00076855000000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.990894+0000","last_change":"2024-09-16T10:32:00.990985+0000","last_active":"2024-09-16T10:32:00.990894+0000","last_peered":"2024-09-16T10:32:00.990894+0000","last_clean":"2024-09-16T10:32:00.990894+0000","last_became_active":"2024-09-16T10:29:30.039431+0000","last_became_peered":"2024-09-16T10:29:30.039431+0000","last_unstale":"2024-09-16T10:32:00.990894+0000","last_undegraded":"2024-09-16T10:32:00.990894+0000","last_fullsized":"2024-09-16T10:32:00.990894+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:00.990876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:32:00.990876+0000","last_clean_scrub_stamp":"2024-09-16T10:32:00.990876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:50:10.425769+0000","scrub_duration":0.001283902,"objects_trimmed":0,"snaptrim_duration":0.00041305900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:32:01.751006+0000","last_change":"2024-09-16T10:32:01.751098+0000","last_active":"2024-09-16T10:32:01.751006+0000","last_peered":"2024-09-16T10:32:01.751006+0000","last_clean":"2024-09-16T10:32:01.751006+0000","last_became_active":"2024-09-16T10:29:30.043748+0000","last_became_peered":"2024-09-16T10:29:30.043748+0000","last_unstale":"2024-09-16T10:32:01.751006+0000","last_undegraded":"2024-09-16T10:32:01.751006+0000","last_fullsized":"2024-09-16T10:32:01.751006+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:01.750984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:32:01.750984+0000","last_clean_scrub_stamp":"2024-09-16T10:32:01.750984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:14:22.762198+0000","scrub_duration":0.0016077070000000001,"objects_trimmed":0,"snaptrim_duration":0.000349687,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"22'2","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:32:01.409404+0000","last_change":"2024-09-16T10:32:01.409513+0000","last_active":"2024-09-16T10:32:01.409404+0000","last_peered":"2024-09-16T10:32:01.409404+0000","last_clean":"2024-09-16T10:32:01.409404+0000","last_became_active":"2024-09-16T10:29:30.038374+0000","last_became_peered":"2024-09-16T10:29:30.038374+0000","last_unstale":"2024-09-16T10:32:01.409404+0000","last_undegraded":"2024-09-16T10:32:01.409404+0000","last_fullsized":"2024-09-16T10:32:01.409404+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'2","last_scrub_stamp":"2024-09-16T10:32:01.409381+0000","last_deep_scrub":"22'2","last_deep_scrub_stamp":"2024-09-16T10:32:01.409381+0000","last_clean_scrub_stamp":"2024-09-16T10:32:01.409381+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:27:01.294823+0000","scrub_duration":0.003499098,"objects_trimmed":0,"snaptrim_duration":0.00040381400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'119","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.716974+0000","last_change":"2024-09-16T10:31:57.717072+0000","last_active":"2024-09-16T10:31:57.716974+0000","last_peered":"2024-09-16T10:31:57.716974+0000","last_clean":"2024-09-16T10:31:57.716974+0000","last_became_active":"2024-09-16T10:29:29.004894+0000","last_became_peered":"2024-09-16T10:29:29.004894+0000","last_unstale":"2024-09-16T10:31:57.716974+0000","last_undegraded":"2024-09-16T10:31:57.716974+0000","last_fullsized":"2024-09-16T10:31:57.716974+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'119","last_scrub_stamp":"2024-09-16T10:31:57.716951+0000","last_deep_scrub":"22'119","last_deep_scrub_stamp":"2024-09-16T10:31:57.716951+0000","last_clean_scrub_stamp":"2024-09-16T10:31:57.716951+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:15:22.634081+0000","scrub_duration":0.0036445779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.502,"15min":0.49199999999999999},"min":{"1min":0.39400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66300000000000003,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.63800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.501},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.47599999999999998,"15min":0.46700000000000003},"min":{"1min":0.40799999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.62}]},{"osd":2,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.35599999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.503,"15min":0.498},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.49199999999999999}]},{"osd":3,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36599999999999999,"15min":0.36499999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.51000000000000001,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.38800000000000001},"min":{"1min":0.312,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Sep 16 10:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.32300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.34499999999999997}]}]},{"osd":2,"up_from":12,"seq":51539607587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.308,"15min":0.308},"min":{"1min":0.23200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.39200000000000002,"5min":0.439,"15min":0.439},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.442,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.20200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.34300000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42399999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.34599999999999997,"15min":0.34799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.32100000000000001}]},{"osd":3,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.504,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.29899999999999999}]},{"osd":4,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.40500000000000003}]},{"osd":5,"last update":"Mon Sep 16 10:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.39400000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640292,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":68745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.627,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.505},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.376,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.42299999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.50900000000000001,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63700000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.495},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.40100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.40100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.626,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.438},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65300000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45100000000000001}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.55200000000000005,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.39100000000000001}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41899999999999998,"15min":0.41999999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54200000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.438,"15min":0.437},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.36699999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640293,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.505,"15min":0.495},"min":{"1min":0.41099999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68400000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.42299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.48099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68000000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.378},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.46200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.433}]},{"osd":2,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.45300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.499},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53500000000000003,"15min":0.52900000000000003},"min":{"1min":0.44900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.53200000000000003}]},{"osd":4,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.32100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.29399999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Sep 16 10:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46700000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":200704,"data_stored":68764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.375,"15min":0.37},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54900000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.43099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34399999999999997},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.497,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.379},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.65500000000000003}]},{"osd":4,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.41099999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.80300000000000005}]},{"osd":5,"last update":"Mon Sep 16 10:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78400000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.754},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.42799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.63900000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640292,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":729088,"data_stored":597673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.32700000000000001},"min":{"1min":0.26800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.44600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.375,"15min":0.379},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.41999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.315}]},{"osd":2,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.26500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.38700000000000001,"15min":0.38900000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.48899999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.747,"15min":0.78200000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68899999999999995,"5min":5.032,"15min":5.032},"last":0.58499999999999996}]},{"osd":4,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67600000000000005,"5min":4.9240000000000004,"15min":4.9240000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.57599999999999996}]},{"osd":5,"last update":"Mon Sep 16 10:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.75700000000000001,"15min":0.79000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.71999999999999997,"5min":4.9800000000000004,"15min":4.9800000000000004},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.753,"5min":5.0030000000000001,"15min":5.0030000000000001},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:18.044 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T10:32:18.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.136+0000 7fb449696640 10 monclient: get_monmap_and_config 2024-09-16T10:32:18.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.136+0000 7fb449696640 10 monclient: build_initial_monmap 2024-09-16T10:32:18.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.136+0000 7fb449696640 10 monclient: monmap: 2024-09-16T10:32:18.139 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:32:18.140 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:18.140 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:32:18.138495+0000 2024-09-16T10:32:18.140 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:32:18.138495+0000 2024-09-16T10:32:18.140 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:32:18.140 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:18.140 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:32:18.140 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:32:18.140 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:32:18.140 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:32:18.141 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:32:18.141 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:32:18.141 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:32:18.141 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:32:18.141 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:32:18.141 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:18.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.137+0000 7fb449696640 1 Processor -- start 2024-09-16T10:32:18.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.137+0000 7fb449696640 1 -- start start 2024-09-16T10:32:18.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.137+0000 7fb449696640 10 monclient: init 2024-09-16T10:32:18.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.137+0000 7fb449696640 10 monclient: _reopen_session rank -1 2024-09-16T10:32:18.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.137+0000 7fb449696640 10 monclient: _add_conns ranks=[0,7,5,6,2,1,8,3,4] 2024-09-16T10:32:18.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.137+0000 7fb449696640 10 monclient(hunting): picked mon.noname-a con 0x7fb4440a2140 addr v1:172.21.15.94:6789/0 2024-09-16T10:32:18.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.137+0000 7fb449696640 10 monclient(hunting): picked mon.noname-i con 0x7fb4440a15e0 addr v1:172.21.15.152:6792/0 2024-09-16T10:32:18.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.138+0000 7fb449696640 10 monclient(hunting): picked mon.noname-h con 0x7fb4440a19f0 addr v1:172.21.15.152:6791/0 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.138+0000 7fb449696640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb444061ae0 con 0x7fb4440a2140 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.138+0000 7fb449696640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4440a0920 con 0x7fb4440a19f0 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.138+0000 7fb449696640 1 -- --> v1:172.21.15.152:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4440a0360 con 0x7fb4440a15e0 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.138+0000 7fb449696640 10 monclient(hunting): _renew_subs 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.138+0000 7fb449696640 10 monclient(hunting): authenticate will time out at 905.479059s 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.138+0000 7fb442ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb4440a2140 0x7fb4440a2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54848/0 (socket says 172.21.15.94:54848) 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.138+0000 7fb442ffd640 1 -- 172.21.15.94:0/1584350513 learned_addr learned my addr 172.21.15.94:0/1584350513 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb442ffd640 1 -- 172.21.15.94:0/1584350513 >> v1:172.21.15.94:6789/0 conn(0x7fb4440a2140 legacy=0x7fb4440a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 -- 172.21.15.94:0/1584350513 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408787579 0 0) 0x7fb444061ae0 con 0x7fb4440a2140 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient(hunting): _negotiate 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:18.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient(hunting): my global_id is 4662 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 -- 172.21.15.94:0/1584350513 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb430003520 con 0x7fb4440a2140 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb43bfff640 1 -- 172.21.15.94:0/1584350513 >> v1:172.21.15.152:6792/0 conn(0x7fb4440a15e0 legacy=0x7fb44413b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4437fe640 1 -- 172.21.15.94:0/1584350513 >> v1:172.21.15.152:6791/0 conn(0x7fb4440a19f0 legacy=0x7fb44413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 -- 172.21.15.94:0/1584350513 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 164349879 0 0) 0x7fb430003520 con 0x7fb4440a2140 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 -- 172.21.15.94:0/1584350513 >> v1:172.21.15.152:6792/0 conn(0x7fb4440a15e0 legacy=0x7fb44413b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 -- 172.21.15.94:0/1584350513 >> v1:172.21.15.152:6791/0 conn(0x7fb4440a19f0 legacy=0x7fb44413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: _finish_hunting 0 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 monclient: found mon.noname-a 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 -- 172.21.15.94:0/1584350513 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4440a0640 con 0x7fb4440a2140 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: _finish_auth 0 2024-09-16T10:32:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: _check_auth_tickets 2024-09-16T10:32:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:32:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 -- 172.21.15.94:0/1584350513 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb434001a60 con 0x7fb4440a2140 2024-09-16T10:32:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb449696640 5 monclient: authenticate success, global_id 4662 2024-09-16T10:32:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb449696640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: dump: 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb449696640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 -- 172.21.15.94:0/1584350513 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb434001d70 con 0x7fb4440a2140 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb449696640 10 monclient: get_monmap_and_config success 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 1 -- 172.21.15.94:0/1584350513 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb434004be0 con 0x7fb4440a2140 2024-09-16T10:32:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb449696640 4 set_mon_vals no callback set 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.140+0000 7fb4427fc640 10 monclient: dump: 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:18.159 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.141+0000 7fb449696640 10 monclient: shutdown 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.141+0000 7fb449696640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.141+0000 7fb449696640 1 -- 172.21.15.94:0/1584350513 >> v1:172.21.15.94:6789/0 conn(0x7fb4440a2140 legacy=0x7fb4440a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.141+0000 7fb449696640 1 -- 172.21.15.94:0/1584350513 shutdown_connections 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.141+0000 7fb449696640 1 -- 172.21.15.94:0/1584350513 >> 172.21.15.94:0/1584350513 conn(0x7fb4440a93c0 msgr2=0x7fb4440a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.141+0000 7fb449696640 1 -- 172.21.15.94:0/1584350513 shutdown_connections 2024-09-16T10:32:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.141+0000 7fb449696640 1 -- 172.21.15.94:0/1584350513 wait complete. 2024-09-16T10:32:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.141+0000 7fb449696640 10 monclient: build_initial_monmap 2024-09-16T10:32:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.141+0000 7fb449696640 10 monclient: monmap: 2024-09-16T10:32:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-09-16T10:32:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:32:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-09-16T10:32:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-09-16T10:32:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-09-16T10:32:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-09-16T10:32:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.noname-f 2024-09-16T10:32:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-09-16T10:32:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.noname-g 2024-09-16T10:32:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.noname-c 2024-09-16T10:32:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.noname-h 2024-09-16T10:32:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.noname-d 2024-09-16T10:32:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.noname-i 2024-09-16T10:32:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.noname-e 2024-09-16T10:32:18.163 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 1 Processor -- start 2024-09-16T10:32:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 1 -- start start 2024-09-16T10:32:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 10 monclient: init 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 10 monclient: _reopen_session rank -1 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 10 monclient: _add_conns ranks=[5,3,1,6,8,4,2,0,7] 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 10 monclient(hunting): picked mon.noname-h con 0x7fb4440a15e0 addr v1:172.21.15.152:6791/0 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 10 monclient(hunting): picked mon.noname-g con 0x7fb4440a19f0 addr v1:172.21.15.152:6790/0 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 10 monclient(hunting): picked mon.noname-f con 0x7fb4440a2140 addr v1:172.21.15.152:6789/0 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 1 -- --> v1:172.21.15.152:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4440a7730 con 0x7fb4440a2140 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 1 -- --> v1:172.21.15.152:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44409f880 con 0x7fb4440a19f0 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 1 -- --> v1:172.21.15.152:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4440a7c60 con 0x7fb4440a15e0 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 10 monclient(hunting): _renew_subs 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.142+0000 7fb449696640 10 monclient(hunting): authenticate will time out at 905.483755s 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.143+0000 7fb442ffd640 1 --1- >> v1:172.21.15.152:6791/0 conn(0x7fb4440a15e0 0x7fb4441cb3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6791/0 says I am v1:172.21.15.94:41394/0 (socket says 172.21.15.94:41394) 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.143+0000 7fb4437fe640 1 --1- >> v1:172.21.15.152:6789/0 conn(0x7fb4440a2140 0x7fb4441d3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.152:6789/0 says I am v1:172.21.15.94:35690/0 (socket says 172.21.15.94:35690) 2024-09-16T10:32:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.143+0000 7fb442ffd640 1 -- 172.21.15.94:0/1799652773 learned_addr learned my addr 172.21.15.94:0/1799652773 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb442ffd640 1 -- 172.21.15.94:0/1799652773 >> v1:172.21.15.152:6791/0 conn(0x7fb4440a15e0 legacy=0x7fb4441cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43bfff640 1 -- 172.21.15.94:0/1799652773 >> v1:172.21.15.152:6790/0 conn(0x7fb4440a19f0 legacy=0x7fb4441d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb4437fe640 1 -- 172.21.15.94:0/1799652773 >> v1:172.21.15.152:6789/0 conn(0x7fb4440a2140 legacy=0x7fb4441d3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.3 v1:172.21.15.152:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451440710 0 0) 0x7fb44409f880 con 0x7fb4440a19f0 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): my global_id is 4953 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb424003520 con 0x7fb4440a19f0 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.5 v1:172.21.15.152:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638681111 0 0) 0x7fb4440a7c60 con 0x7fb4440a15e0 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:32:18.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): my global_id is 4775 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb44409f880 con 0x7fb4440a15e0 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981513405 0 0) 0x7fb4440a7730 con 0x7fb4440a2140 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _negotiate 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): my global_id is 4681 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4440a7c60 con 0x7fb4440a2140 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4254395312 0 0) 0x7fb4440a7c60 con 0x7fb4440a2140 2024-09-16T10:32:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4440a7730 con 0x7fb4440a2140 2024-09-16T10:32:18.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.3 v1:172.21.15.152:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3954541417 0 0) 0x7fb424003520 con 0x7fb4440a19f0 2024-09-16T10:32:18.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4440a7c60 con 0x7fb4440a19f0 2024-09-16T10:32:18.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb43c002050 con 0x7fb4440a2140 2024-09-16T10:32:18.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:18.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-16T10:32:18.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): dump: 2024-09-16T10:32:18.167 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:18.168 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:18.168 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.168 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.168 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:18.168 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:18.168 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:18.169 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:18.169 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:18.169 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:18.169 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:18.169 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:18.170 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:18.170 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:18.170 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:18.170 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:18.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T10:32:18.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.5 v1:172.21.15.152:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1108120895 0 0) 0x7fb44409f880 con 0x7fb4440a15e0 2024-09-16T10:32:18.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb424003520 con 0x7fb4440a15e0 2024-09-16T10:32:18.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.3 v1:172.21.15.152:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb42c001cd0 con 0x7fb4440a19f0 2024-09-16T10:32:18.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:18.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-16T10:32:18.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.145+0000 7fb43b7fe640 10 monclient(hunting): dump: 2024-09-16T10:32:18.172 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:18.172 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:18.172 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.172 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.172 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:18.172 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:18.172 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:18.172 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.5 v1:172.21.15.152:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb434004b20 con 0x7fb4440a15e0 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient(hunting): dump: 2024-09-16T10:32:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:18.174 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:18.174 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.174 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.174 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:18.174 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:18.174 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:18.174 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4005533809 0 0) 0x7fb4440a7730 con 0x7fb4440a2140 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 >> v1:172.21.15.152:6791/0 conn(0x7fb4440a15e0 legacy=0x7fb4441cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 >> v1:172.21.15.152:6790/0 conn(0x7fb4440a19f0 legacy=0x7fb4441d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient: _finish_hunting 0 2024-09-16T10:32:18.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 1 monclient: found mon.f 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44413faa0 con 0x7fb4440a2140 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient: _finish_auth 0 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient: _check_auth_tickets 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb449696640 5 monclient: authenticate success, global_id 4681 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb449696640 10 monclient: _renew_subs 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb449696640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44413f7c0 con 0x7fb4440a2140 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb449696640 10 monclient: _renew_subs 2024-09-16T10:32:18.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb449696640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44413e530 con 0x7fb4440a2140 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb43c003c40 con 0x7fb4440a2140 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 3301529438 0 0) 0x7fb43c0040f0 con 0x7fb4440a2140 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb441ffb640 4 set_mon_vals no callback set 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.146+0000 7fb43b7fe640 10 monclient: dump: 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:fsid f51dbe5c-da09-4bb3-8cd3-867d68869245 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.177 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-09-16T10:28:46.257482+0000 2024-09-16T10:32:18.178 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 18 (reef) 2024-09-16T10:32:18.178 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-09-16T10:32:18.178 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-09-16T10:32:18.178 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.152:6789/0 mon.f 2024-09-16T10:32:18.178 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.b 2024-09-16T10:32:18.178 INFO:teuthology.orchestra.run.smithi094.stderr:3: v1:172.21.15.152:6790/0 mon.g 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr:4: v1:172.21.15.94:6791/0 mon.c 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr:5: v1:172.21.15.152:6791/0 mon.h 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr:6: v1:172.21.15.94:6792/0 mon.d 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr:7: v1:172.21.15.152:6792/0 mon.i 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr:8: v1:172.21.15.94:6793/0 mon.e 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.147+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 7 ==== mgrmap(e 5) v1 ==== 44782+0+0 (unknown 3905464607 0 0) 0x7fb43c004250 con 0x7fb4440a2140 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.147+0000 7fb449696640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.147+0000 7fb449696640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:32:18.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.147+0000 7fb449696640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:32:18.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.147+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb428005040 con 0x7fb4440a2140 2024-09-16T10:32:18.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.147+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3506+0+0 (unknown 1869123988 0 0) 0x7fb43c0460d0 con 0x7fb4440a2140 2024-09-16T10:32:18.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.147+0000 7fb442ffd640 10 monclient: get_auth_request con 0x7fb424039d20 auth_method 0 2024-09-16T10:32:18.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.152+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (unknown 1092875540 0 801140998) 0x7fb43c0155e0 con 0x7fb4440a2140 2024-09-16T10:32:18.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.152+0000 7fb43b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:32:18.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.152+0000 7fb43b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T10:32:18.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.312+0000 7fb449696640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T10:32:18.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.312+0000 7fb449696640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T10:32:18.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.312+0000 7fb449696640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.152:6789/0 2024-09-16T10:32:18.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:18.312+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 --> v1:172.21.15.152:6789/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fb428005870 con 0x7fb4440a2140 2024-09-16T10:32:19.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.025+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 10 ==== config(1 keys) v1 ==== 35+0+0 (unknown 2555425225 0 0) 0x7fb43c0155e0 con 0x7fb4440a2140 2024-09-16T10:32:19.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.025+0000 7fb43b7fe640 10 monclient: handle_config config(1 keys) v1 2024-09-16T10:32:19.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.025+0000 7fb441ffb640 4 set_mon_vals no callback set 2024-09-16T10:32:19.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.025+0000 7fb441ffb640 10 set_mon_vals mon_health_to_clog = false 2024-09-16T10:32:19.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.027+0000 7fb43b7fe640 1 -- 172.21.15.94:0/1799652773 <== mon.1 v1:172.21.15.152:6789/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) v1 ==== 123+0+0 (unknown 4099060384 0 0) 0x7fb43c013a50 con 0x7fb4440a2140 2024-09-16T10:32:19.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.027+0000 7fb43b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T10:32:19.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.027+0000 7fb43b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T10:32:19.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.029+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 >> v1:172.21.15.152:6800/2686360781 conn(0x7fb424039d20 legacy=0x7fb42403c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:19.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.029+0000 7fb449696640 10 monclient: shutdown 2024-09-16T10:32:19.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.029+0000 7fb449696640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:32:19.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.029+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 >> v1:172.21.15.152:6789/0 conn(0x7fb4440a2140 legacy=0x7fb4441d3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:32:19.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.029+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 shutdown_connections 2024-09-16T10:32:19.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.029+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 >> 172.21.15.94:0/1799652773 conn(0x7fb4440a93c0 msgr2=0x7fb4440a7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:32:19.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.030+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 shutdown_connections 2024-09-16T10:32:19.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-16T10:32:19.030+0000 7fb449696640 1 -- 172.21.15.94:0/1799652773 wait complete. 2024-09-16T10:32:19.045 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T10:32:19.046 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T10:32:19.046 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T10:32:19.046 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.075 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T10:32:19.075 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T10:32:19.075 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.150 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T10:32:19.150 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T10:32:19.150 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.213 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T10:32:19.214 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T10:32:19.214 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.249 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T10:32:19.250 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T10:32:19.250 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.317 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T10:32:19.317 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T10:32:19.317 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.390 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T10:32:19.390 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T10:32:19.391 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T10:32:19.391 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.460 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T10:32:19.460 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T10:32:19.460 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T10:32:19.460 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.473 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T10:32:19.473 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T10:32:19.473 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.529 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T10:32:19.529 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T10:32:19.529 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.582 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T10:32:19.582 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-16T10:32:19.582 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.634 INFO:tasks.ceph.mon.d:Stopped 2024-09-16T10:32:19.635 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-09-16T10:32:19.635 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.686 INFO:tasks.ceph.mon.e:Stopped 2024-09-16T10:32:19.686 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-09-16T10:32:19.687 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.700 INFO:tasks.ceph.mon.f:Stopped 2024-09-16T10:32:19.700 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-09-16T10:32:19.700 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.753 INFO:tasks.ceph.mon.g:Stopped 2024-09-16T10:32:19.753 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-09-16T10:32:19.753 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.805 INFO:tasks.ceph.mon.h:Stopped 2024-09-16T10:32:19.805 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-09-16T10:32:19.805 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:32:19.857 INFO:tasks.ceph.mon.i:Stopped 2024-09-16T10:32:19.858 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T10:32:19.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | head -n 1 2024-09-16T10:32:19.891 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-16T10:32:19.892 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T10:32:20.061 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-16T10:32:20.062 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T10:32:20.145 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-16T10:32:20.146 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T10:32:20.220 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:32:20.220 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T10:32:20.356 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:32:20.357 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T10:32:20.445 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:32:20.445 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T10:32:20.530 INFO:tasks.ceph:Archiving mon data... 2024-09-16T10:32:20.532 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/data/mon.a.tgz 2024-09-16T10:32:20.532 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-09-16T10:32:20.550 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.KgjuH1a7tH 2024-09-16T10:32:20.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.KgjuH1a7tH 2024-09-16T10:32:20.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.KgjuH1a7tH 2024-09-16T10:32:20.736 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.KgjuH1a7tH is 106KB 2024-09-16T10:32:20.793 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.KgjuH1a7tH 2024-09-16T10:32:20.808 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/data/mon.b.tgz 2024-09-16T10:32:20.809 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-09-16T10:32:20.864 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.n4UMRgRsPw 2024-09-16T10:32:20.864 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.n4UMRgRsPw 2024-09-16T10:32:20.947 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.n4UMRgRsPw 2024-09-16T10:32:21.069 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.n4UMRgRsPw is 106KB 2024-09-16T10:32:21.120 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.n4UMRgRsPw 2024-09-16T10:32:21.135 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/data/mon.c.tgz 2024-09-16T10:32:21.135 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-09-16T10:32:21.189 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.q2rHNXcN4j 2024-09-16T10:32:21.190 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.q2rHNXcN4j 2024-09-16T10:32:21.271 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.q2rHNXcN4j 2024-09-16T10:32:21.395 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.q2rHNXcN4j is 106KB 2024-09-16T10:32:21.446 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.q2rHNXcN4j 2024-09-16T10:32:21.460 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/data/mon.d.tgz 2024-09-16T10:32:21.460 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-09-16T10:32:21.515 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.0Dao3yZmT3 2024-09-16T10:32:21.516 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.0Dao3yZmT3 2024-09-16T10:32:21.597 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.0Dao3yZmT3 2024-09-16T10:32:21.720 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.0Dao3yZmT3 is 104KB 2024-09-16T10:32:21.772 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.0Dao3yZmT3 2024-09-16T10:32:21.787 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/data/mon.e.tgz 2024-09-16T10:32:21.787 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-09-16T10:32:21.841 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.fKNjQkH4Yg 2024-09-16T10:32:21.842 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.fKNjQkH4Yg 2024-09-16T10:32:21.922 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.fKNjQkH4Yg 2024-09-16T10:32:22.046 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.fKNjQkH4Yg is 106KB 2024-09-16T10:32:22.097 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.fKNjQkH4Yg 2024-09-16T10:32:22.112 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/data/mon.f.tgz 2024-09-16T10:32:22.112 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-16T10:32:22.131 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.Utq2N5YcDL 2024-09-16T10:32:22.131 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.Utq2N5YcDL 2024-09-16T10:32:22.354 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.Utq2N5YcDL 2024-09-16T10:32:22.442 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.Utq2N5YcDL is 106KB 2024-09-16T10:32:22.494 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.Utq2N5YcDL 2024-09-16T10:32:22.509 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/data/mon.g.tgz 2024-09-16T10:32:22.509 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-16T10:32:22.565 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.elgaukLGYr 2024-09-16T10:32:22.565 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.elgaukLGYr 2024-09-16T10:32:22.649 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.elgaukLGYr 2024-09-16T10:32:22.774 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.elgaukLGYr is 106KB 2024-09-16T10:32:22.824 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.elgaukLGYr 2024-09-16T10:32:22.838 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/data/mon.h.tgz 2024-09-16T10:32:22.839 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-16T10:32:22.894 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.VC4Dlq12u7 2024-09-16T10:32:22.894 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.VC4Dlq12u7 2024-09-16T10:32:22.977 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.VC4Dlq12u7 2024-09-16T10:32:23.101 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.VC4Dlq12u7 is 106KB 2024-09-16T10:32:23.152 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.VC4Dlq12u7 2024-09-16T10:32:23.167 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/data/mon.i.tgz 2024-09-16T10:32:23.167 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-16T10:32:23.221 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.q08dEwceYD 2024-09-16T10:32:23.222 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.q08dEwceYD 2024-09-16T10:32:23.303 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.q08dEwceYD 2024-09-16T10:32:23.427 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.q08dEwceYD is 106KB 2024-09-16T10:32:23.478 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.q08dEwceYD 2024-09-16T10:32:23.493 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T10:32:23.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T10:32:23.497 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T10:32:23.557 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T10:32:23.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T10:32:23.566 DEBUG:teuthology.orchestra.run.smithi152:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T10:32:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T10:32:23.624 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T10:32:23.628 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T10:32:23.630 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/remote/smithi094/crash 2024-09-16T10:32:23.631 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T10:32:23.669 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/remote/smithi152/crash 2024-09-16T10:32:23.670 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T10:32:23.694 INFO:tasks.ceph:Compressing logs... 2024-09-16T10:32:23.695 DEBUG:teuthology.orchestra.run.smithi094:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T10:32:23.714 DEBUG:teuthology.orchestra.run.smithi152:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T10:32:26.558 INFO:tasks.ceph:Archiving logs... 2024-09-16T10:32:26.559 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/remote/smithi094/log 2024-09-16T10:32:26.560 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T10:32:27.021 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/remote/smithi152/log 2024-09-16T10:32:27.021 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T10:32:27.187 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T10:32:27.197 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T10:32:27.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T10:32:27.199 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T10:32:27.277 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T10:32:27.286 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T10:32:27.286 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:32:27.288 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:32:27.301 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:32:27.305 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:32:27.306 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:32:27.306 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-09-16T10:32:27.306 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv01.front.sepia.ceph.com 3 6 377 26 -618us[ -615us] +/- 50ms 2024-09-16T10:32:27.306 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 24 +1308us[+1308us] +/- 55ms 2024-09-16T10:32:27.306 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -765us[ -765us] +/- 75ms 2024-09-16T10:32:27.306 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:32:27.310 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:32:27.310 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-09-16T10:32:27.310 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv01.front.sepia.ceph.com 3 6 377 24 -563us[ -558us] +/- 50ms 2024-09-16T10:32:27.310 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 +1306us[+1312us] +/- 55ms 2024-09-16T10:32:27.310 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -749us[ -743us] +/- 75ms 2024-09-16T10:32:27.310 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:32:27.310 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T10:32:27.318 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T10:32:27.319 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T10:32:27.425 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T10:32:27.489 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T10:32:27.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:32:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:27.586 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:32:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:27.616 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T10:32:27.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T10:32:27.668 INFO:teuthology.task.internal:Duration was 579.505544 seconds 2024-09-16T10:32:27.669 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T10:32:27.712 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T10:32:27.712 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T10:32:27.714 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T10:32:27.751 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:32:27.756 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:32:28.105 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T10:32:28.105 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-09-16T10:32:28.105 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T10:32:28.166 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T10:32:28.166 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T10:32:28.196 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T10:32:28.196 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T10:32:28.209 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T10:32:28.268 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T10:32:28.268 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T10:32:28.275 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T10:32:28.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T10:32:28.420 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T10:32:28.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T10:32:28.445 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T10:32:28.471 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T10:32:28.484 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:32:28.487 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:32:28.508 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-09-16T10:32:28.535 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-09-16T10:32:28.578 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:32:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.599 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:32:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T10:32:28.645 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T10:32:28.646 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/remote/smithi094 2024-09-16T10:32:28.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T10:32:28.680 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906655/remote/smithi152 2024-09-16T10:32:28.680 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T10:32:28.713 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T10:32:28.713 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T10:32:28.716 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T10:32:28.764 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T10:32:28.775 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T10:32:28.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T10:32:28.822 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T10:32:28.823 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T10:32:28.825 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T10:32:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 10:32 /home/ubuntu/cephtest 2024-09-16T10:32:28.840 INFO:teuthology.orchestra.run.smithi152.stdout: 265340 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 10:32 /home/ubuntu/cephtest 2024-09-16T10:32:28.841 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T10:32:28.860 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T10:32:28.942 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} duration: 579.5055441856384 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T10:32:28.942 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T10:32:29.017 INFO:teuthology.run:pass